LazyMethodsEnabler tool

Following hypervisor.fr’s post Storage vMotion : the method is disabled, we work today for a new customer whom had too many some virtual machine with disabled methods (and increasing on a daily basis thanks to backup jobs…) :

Without blaming existing yellow and black crappy backup tool (although it is quite tempting…), there are too many affected virtual machines to fit any of these VMware KB workarounds. As a reminder, workarounds are:

  • VM unregister/register (bye bye statistics, custom fields and history as MoRef is no longer the same > NoWay)
  • vCenter’s database’s manual purge (this operation needs vCenter service to be stopped, so it’s not really a production workaround if it occurs often > NoWay)
  • Restart a backup job (that is viable on a small scale, but in our case, the number of VM with this problem increased by about 60 daily … > NoWay)

Pending the migration to a real backup solution (the transition to Veeam Backup being expected), we still had to find a cleaner solution :p

Assuming that a restart of the VM backup is likely to resolve the issue, we wondered what is causing the error. The KB explains:

When a VM-level backup begins, the backup system informs vCenter to disable Storage vMotion for that VM to ensure that the backups can complete successfully

The interesting part is “the backup system informs vCenter“. After some research, we finally found this is done through Virtual Disk Development Kit (VDDK) calls. In order to have a better understanding of backup and VMDK files handling by third party tools, we started to read the Virtual Disk Programming Guide looking for some answers.

Besides the extremely interesting part of this SDK (we also recommand you to take a look just to see how amazing it is), what we were looking for is located in Advanced Transport APIs on “Prepare For Access and End Access”.

Here is the excerpt answering our question:

The VixDiskLib_PrepareForAccess() function notifies a vCenter-managed host that a virtual machine’s disks are being opened, probably for backup, so the host should postpone virtual machine operations that might interfere with virtual disk access. Call this function before creating a snapshot on a virtual machine. Internally, this function disables the vSphere API method RelocateVM_Task.


The VixDiskLib_EndAccess() function notifies the host that a virtual machine’s disks have been closed, so operations that rely on the virtual disks to be closed, such as vMotion, can now be allowed. Call this function after closing all the virtual disks, and after deleting the virtual machine snapshot. Normally this function is called after previously calling VixDiskLib_PrepareForAccess, but you can call it to clean up after a crash. Internally, this function re-enables the vSphere API method RelocateVM_Task.

So it’s all about VixDiskLib_PrepareForAccess() and VixDiskLib_EndAccess()  calls who notifies vCenter to handle the disabled methods (restraining for instance to order a Storage vMotion during a backup).

Regarding our issue, restarting a backup job perform the following operations PrepareForAccess>Backup>EndAccess and therefore is supposed to fix the issue thanks to the EndAccess() call. On this basis, we thought that a viable large-scale solution (other than switching to real backup solution) would be to have a program that does the following operations PrepareForAccess>EndAccess (and hence would be much faster than a full backup of each VM).

As VDDK is a collection of C/C++ libraries, we had to bring back some memories on C++ developement skills (we are indeed far from C# or PowerShell…). So we develop a program that will help “reactivate” the methods disabled by a f… tool

This tool is standalone for now, so you must use the VM’s MoRef in argument (yes I know it’s not user friendly, but hey what did you expect, it’s C++ !). It takes the following parameters:

lazyMethodEnabler.exe -host 10.69.69.69 -user "vmdude" -password "esxi4ever" -vm "moid=vm-69"
  • -host <host IP> : vCenter server IP
  • -user <vcenter Username> : username used to connect to vCenter
  • -password <vcenter Password> : username password
  • -vm “moid=moref” : VM’s MoRef

We added an optionnal parameter -disableMethods which allow to disable the RelocateVM_Task method. Used with this parameter, the program will only call VixDiskLib_PrepareForAccess() (ie it will not call VixDiskLib_EndAccess() ) and will set a VM with the disabled method.

Warning, the -disableMethods parameter should only be used for debug purpose!

The Visual Studio 2012 (Visual C++) sources are available here:

lazyMethodsEnabler.zip : only the program

lazyMethodsEnabler-VDDK.zip : program and VDDK5.0 librairies

lazyMethodsEnabler-Sources.zip : Visual C++ sources

Since this program is based on VDDK calls, the SDK must be installed on the same computer as lazyMethodEnabler will be run. Tests were made with VDDK 5.0 (available here : http://www.vmware.com/download/download.do?downloadGroup=VDDK50U2) on a Windows 2008 R2 server.

PS: Sorry Samir it took so long to publish this requested post :p Mea Culpa !

8 comments

  1. Just wanted to say thanks for publishing your executable for enabling svmotion. It has been a huge help. I have taken your app and wrapped into a powershell function, which has made in work quite well in my other scripts. Just need to drop your app and all the dlls into the same folder as the script.

    function Enable-SvMotion
    {

    Param (
    [Parameter (
    Mandatory=$false,
    ValueFromPipeline=$true,
    ParameterSetName=”VMObject”
    )
    ]
    [VMware.VimAutomation.ViCore.Impl.V1.Inventory.VirtualMachineImpl]$VM,
    [Parameter (
    Mandatory=$false,
    ValueFromPipeline=$false,
    ParameterSetName=”VMName”
    )
    ]
    [String]$Name,
    [Parameter (
    Mandatory=$true,
    ValueFromPipeline=$false
    )
    ]
    [System.Management.Automation.PSCredential]$Credential
    )

    BEGIN
    {
    #Set path of app
    $ScriptPath = “$PSScriptRoot\lazyMethodsEnabler.exe”
    }
    PROCESS
    {

    #Get VM object from name
    if ($Name)
    {
    $VM = Get-VM -Name $Name
    }
    foreach ($VMObject in $VM)
    {
    if ($VMObject | Where-Object {$_.ExtensionData.disabledmethod -eq ‘RelocateVM_Task’})
    {
    #Enabling SvMotion
    $Params = @(“-host $($VMObject.ExtensionData.Client.ServiceUrl.split(‘/’)[2])”,
    “-user $($Credential.UserName)”, “-password $(‘”‘+$Credential.GetNetworkCredential().password+'”‘)”, “-vm $(‘”‘+$VMObject.Id.Replace(‘VirtualMachine-‘,’moid=’)+'”‘)”)
    Invoke-Expression “$ScriptPath $Params” | Out-Null
    $VMObject
    }
    else
    {
    #Write-Host -ForegroundColor Yellow $VMObject.name “already enabled”
    }
    }
    }
    END
    {

    }
    }

    • Thanks for your feedback, I’m happy it could be useful !
      And thanks for your script.

      I must finish a post where I explain how to schedule this tool with PowerCLI script in order to re-enable methods for all VM that are not in current backup state.

  2. I’m a real newbie in VM and don’t have scripting/coding capabilities, but I ran into the same problem this article adresses. So I’ve got some perhaps stupid questions so bear with me 🙂

    1. We’re using ESX-5. Will it work?
    2. Can it make any damage what so ever to the VM? Do I need to take some kind of backup before running this app?
    3. Do I run this app from a standard command line? or from a Powershell window?

    • 1. This tool is based on VDDK 5.0 so ESX 5 is supported
      2. The only thing it does it to send some command (PrepareAccess() and/or EndAccess() ) through VDDK API calls so it’s harmless
      3. The executable provided is compiled so you can run it from command line or powershell shell

  3. Any chance to get a version updated for vddk 6.5? Having the same issues with 6 and 6.5 environments and have so far been unable to get this executable working with versions of the vddk that can connect to more recent vsphere editions..

Leave a Reply

Required fields are marked *.