While we were looking to reduce execution time of some script (or should I say OneLiner), we found a new way for script enhancement, in the same spirit as NoSQL, based on a key-value association rather than a complex relational schema.

This is done using hash tables that will be populated at the beginning of the script and be used afterward. The hash table structure is very useful for search operations, because the time consumption is very low, with an average of O(1) in BigO notation (aka no time wasted):

For instance, we had a relatively simple OneLiner which listed virtual machines that were not in a resource pool (ie with ‘Resources’ resource pool as parent, you can find more information on page 43 of VMware vSphere Resource Management document) and their cluster:

get-view -ViewType virtualmachine -Property ResourcePool, Name | ?{(Get-View $_.ResourcePool -Property Name).Name -eq "Resources"} | Select @{n="Cluster";e={(get-view (Get-View $_.ResourcePool -Property Parent).Parent -Property Name).Name}}, Name

This OneLiner filters already all Get-View calls to minimize execution time but is still taking a lot of time…

So we tried to look into hash tables in order to see what could be done with it. We created several hash tables in order to get rid of Get-View calls (except of course for the first one retrieving virtual machines).

$htabResourcePool = @{}
$htabResourcePoolParent = @{}
$htabCluster = @{}

Once hash tables were declared, we filled them up with needed data (using MoRef as key) :

get-view -viewtype resourcepool -property name -Filter @{"name"="Resources"} | %{$htabResourcePool.Add($_.MoRef,$_.Name)}
get-view -viewtype resourcepool -property parent -Filter @{"name"="Resources"} | %{$htabResourcePoolParent.Add($_.MoRef,$_.Parent)}
get-view -viewtype clustercomputeresource -property name | %{$htabCluster.Add($_.MoRef,$_.Name)}

Finally we use these tables in the OneLiner, replacing Get-View calls by hash tables search operations:

get-view -ViewType virtualmachine -Property ResourcePool, Name | ?{$htabResourcePool[$_.ResourcePool] -eq "Resources"} | Select @{n="Cluster";e={$htabCluster[$htabResourcePoolParent[$_.ResourcePool]]}}, Name

For now, we reach the same results as the method without hash tables but now we have to look into the execution time :p

So we created a sample script which will run process using both methods:

1. the first method will use Get-View calls only
2. the second method will populate hash tables and will use them afterward (the hash table filling will be part of the process to fit the exact same perimeter).

The script will display the two methods’ results’ count (to ensure the similarity of the returned content) also with the duration. In our example, we had a little less than 2200 VMs on the platform and here are the results:

In order to not mess with hypervisor.fr (we’re already hearing him shouting “Don’t touch my OneLiner!”, the chosen example is actually very meaningful because of the Get-View calls overlaping so the transition to hash tables were very effective :p

It is possible though that depending on the script used, the gain of using hash tables will not be as obvious as it were in this example. However, we invite you to test and compare the execution time.

Here is the script used to perform benchmark using both methods:

Write-Host -ForegroundColor Yellow "Benchmarking starting..."

Write-Host "`nMethod 1 (with regular filtered Get-View)"
$startMethod1 = Get-Date
$resultMethod1 = get-view -ViewType virtualmachine -Property ResourcePool, Name | ?{(Get-View $_.ResourcePool -Property Name).Name -eq "Resources"} | Select @{n="Cluster";e={(get-view (Get-View $_.ResourcePool -Property Parent).Parent -Property Name).Name}}, Name
$endMethod1 = Get-Date

Write-Host -ForegroundColor Green "Found"(($resultMethod1 | Measure-Object).Count)"records in"(($endMethod1 - $startMethod1).TotalSeconds)"seconds"

Write-Host "`nMethod 2 (with hashtable)"
$startMethod2 = Get-Date
$htabResourcePool = @{}
$htabResourcePoolParent = @{}
$htabCluster = @{}

get-view -viewtype resourcepool -property name -Filter @{"name"="Resources"} | %{$htabResourcePool.Add($_.MoRef,$_.Name)}
get-view -viewtype resourcepool -property parent -Filter @{"name"="Resources"} | %{$htabResourcePoolParent.Add($_.MoRef,$_.Parent)}
get-view -viewtype clustercomputeresource -property name | %{$htabCluster.Add($_.MoRef,$_.Name)}

$resultMethod2 = get-view -ViewType virtualmachine -Property ResourcePool, Name | ?{$htabResourcePool[$_.ResourcePool] -eq "Resources"} | Select @{n="Cluster";e={$htabCluster[$htabResourcePoolParent[$_.ResourcePool]]}}, Name
$endMethod2 = Get-Date

Write-Host -ForegroundColor Green "Found"(($resultMethod2 | Measure-Object).Count)"records in"(($endMethod2 - $startMethod2).TotalSeconds)"seconds"

We would like to thanks again all the readers for everything and give a big hurray to John Troyer for the vExpert program and what it means!

I will not detail all the project history (you can get it in the must-read posts from William Lam vOpenData: An Open Virtualization Community Database or from Ben Thomas vOpenData – Crunching Everyone’s Data For Fun And Knowledge), but basically, vOpenData is a community based statistics database (giving us some answers for eternal questions as “What is the average size of VMDK?” or “What is the average consolidation ratio?”)

The project is composed with a small script (Perl or PowerCLI, pick one!) available from GitHub (https://github.com/vopendata/scripts) which will gather some statistics from your environnement (fully anonymously, there is no name, only UUID are reported), and the www.vopendata.org website which will let you upload the gathered data.

In the end, the result is just amazing, in just a few minute you’ll have uploaded your stats which will be displayed on a huge dashboard (combining all data gathered from community) available on http://dash.vopendata.org/public :

As you guessed out, all the project is based on community effort, so we count on you to participate, come visit and support www.vopendata.org !

Get-View -ViewType HostSystem -Property Name,Config | Select Name, @{n="IP";e={$_.config.network.vnic.spec.ip.ipaddress}}

Or you can use DNS resolve:

Get-View -ViewType HostSystem -Property Name | Select Name, @{n="IP";e={[System.Net.Dns]::GetHostAddresses($_.Name)}}

Even if the results are the same, execution time differ widely. With ~200 serveurs ESXi, the vSphere SDK method takes around 26s to complete (even with the use of Property filter for the Get-View cmdlet) instead of 0,3s for the DNS one:

Alan Renouf published a post about it, explaining what’s new with Distributed Switches cmdlets: PowerCLI 5.1 R2 Released

As we were reading the release note (available here, there is always good information in it!), we finally saw Powershell v3 support for PowerCLI !!!

This will allow a lot of good stuff, including performance enhancement of Get-ChildItem cmdlet. We will make further tests in order to try again the use of PSDrive in order to get some VM files (like .vmx).

We also saw new way for server connection (this method is quite the same VIClient plugin used) :

vSphere PowerCLI introduces an improvement in PowerCLI views.
With the VimClient.Connect() method, you can now connect to a server by server session ID.

Finally, this release is awesome, let’s play ^^ The update is available on the VMware website: http://blogs.vmware.com/vipowershell/2013/02/powercli-5-1-release-2-now-available.html

Thinking about it, what we wanted to do is quite similar as the Get-View behavior, for instance the possibility to use it like:

Get-View -ViewType VirtualMachines
Get-VM | Get-View
Get-View -ViewType HostSystem
Get-VMHost | Get-View

Thanks to the proxies commands, we looked at the Get-View cmdlet definition, for instance with the following command:

[System.Management.Automation.ProxyCommand]::create((New-Object System.Management.Automation.CommandMetaData(Get-Command Get-View)))

Our PowerCLI function will display graphical ASCII art load in console for several object types (virtual machine, ESX host or vSphere cluster). This function can be used in 2 ways, first as standalone cmdlet:

Get-Load -LoadType VirtualMachine
Get-Load -LoadType HostSystem
Get-Load -LoadType ClustercomputeResource

Then you can pass objects to it through pipeline:

Get-VM "vm*" | Get-Load
Get-VMHost | Get-Load
Get-Cluster -Location "folder01" | Get-Load

Right now, there is only 3 object types supported (it’ll be remember to you if you try to think out the box :p )

Here are samples for ASCII art load for different objects:

It’s been a while we wanted to make a PowerCLI fancy display for vSphere cluster load, in a kind of way VIClient does for server load:

So here is a PowerCLI script that will display your cluster load in CLI:

We had as much fun making the graphical ascii art display as getting the statistics data (especially with the quest of perfect character in order to display filling bar ^^). This PowerCLI script is a Get-ClusterLoad function that took some non-mandatory parameters:

.PARAMETER ClusterName
 The cluster name or names separated by coma
 .PARAMETER Quickstat
 Switch, when true the method to get stats is based on quickstats through summary child properties. If not, the method will use PerfManager instance with QueryPerf method in order to get non computed stats. The default for this switch is $true.

So this function can be used to display the load of all your clusters (default behavior with no argument):

Get-ClusterLoad

You can also display specific cluster load (by giving cluster name separated by comma):

Get-ClusterLoad -ClusterName vCluster01,vCluster02

You can also force reals stats usage instead of quick stats:

Get-ClusterLoad -Quickstat:$false

The 2 ways of retrieving statistics data are pretty much different regarding execution time. First of all, the default method will retrieve quick stats available with runtime.memory and runtime.cpu properties from cluster root resource pool (named Resources, with ManagedObjectReference:ResourcePool type) :

This method is fast as it’ll only retrieve already computed values in order to get cluster load (just a few seconds for almost 200 ESXi servers):

The second method (using -QuickStat:$false switch) will retrieve mem.consumed.average and cpu.usagemhz.average statistic counters from each cluster. This load will be an average one (so it can be slightly different from the load generated with the first method) and will be longer to compute (a little more than 2 minutes for almost 200 ESXi servers):

This function is not extremely useful as-is (except for the fun part with ASCII art ^^), but we’re working on a plugin that could be added to vCheck in order to have these information in the sent report. We’ll update this post when it’ll be finished.

The function displaying graph bar with  percentage input is the sub-function Show-PercentageGraph (line 37 to 50):

function Show-PercentageGraph([int]$percent, [int]$maxSize=20) {
	if ($percent -gt 100) { $percent = 100 }
	$warningThreshold = 60 # percent
	$alertThreshold = 80 # percent
	[string]$g = [char]9632 #this is the graph character, use [char]9608 for full square character
	if ($percent -lt 10) { write-host -nonewline "0$percent [ " } else { write-host -nonewline "$percent [ " }
	for ($i=1; $i -le ($barValue = ([math]::floor($percent * $maxSize / 100)));$i++) {
		if ($i -le ($warningThreshold * $maxSize / 100)) { write-host -nonewline -foregroundcolor darkgreen $g }
		elseif ($i -le ($alertThreshold * $maxSize / 100)) { write-host -nonewline -foregroundcolor yellow $g }
		else { write-host -nonewline -foregroundcolor red $g }
	}
	for ($i=1; $i -le ($traitValue = $maxSize - $barValue);$i++) { write-host -nonewline "-" }
	write-host -nonewline " ]"
}

You can modify the graph width in the sub-function Show-PercentageGraph (line 37):

[int]$maxSize=20

You can modify the threshold for color change for warning/alert in the sub-function Show-PercentageGraph (line 39/40):

$warningThreshold = 60 # percent
$alertThreshold = 80 # percent

We received theses alerts as there was no VM hosted on theses servers, so obviously no network load. As every troubleshooting should use esxtop, we ran it and used the network context (key ‘n’):

What was weird is that we didn’t saw any Transmit Dropped Packet (%DRPTX) or Received Dropped Packet (%DRPRX) while vCenter performance graph displayed a lot of them.

The vCenter counter is a summation one (i.e. a computed counter based on values retrieved from a time range), we wanted to check that esxtop counters stays null. So we ran esxtop in batch mode and redirect the results in a CSV file, and by importing it in perfmon, we were able to confirm that these counters stayed at 0…

We spare you all the failed tests about finding the reason for the misinterpretation between esxtop counters and vCenter performance graph. In an ultimate try in order to understand the origin of this issue, we captured some network traffic on an ESXi server (there was still no VM running on this server and was in maintenance mode) using the command tcpdump-uw :

tcpdump-uw -i vmk0 -s 1514 -w traffic.pcap

This command will run some packet capture, listening on vmk0 interface (-i vmk0), saving all frames (-s 1514 for regular packet, or -s 9014 with Jumbo Frames) in pcap format in order to be able to analyze them in WireShark.

Unfortunately, nothing seems to be strange/wrong or with bad destination which could explain why they’re dropped by vmnic…

Last but not least, as HP ESXi image was quite new (just a few weeks), we thought may be embedded driver could be troublesome. OK they were supported by HP, but we had some painful past experiences with Flex10 driver resulting in PSOD or lost connection, so why not :p

Using command ethtool on blades already in production and on new ones (with the same hardware configuration), we did saw differences regarding driver version (firmware was the same though). On older blades, version was 2.102.518.0 :

Since on new blades, version was 4.1.334.0 :

So we decided to downgrade ESXi driver for theses FlexFabric onboard card in order to get the same driver version as the old blades to be able to compare.

You can do it with the esxupdate command, first, you have to get the installed package name:

esxupdate query --vib-view | grep -i be2net

Then, you have to delete this package:

esxupdate -b cross_oem-vmware-esx-drivers-net-be2net_400.4.1.334.0-1vmw.2.17.249663 remove

Finally, you install requested version (the bundle has been uploaded on a datastore in order to have an easy access from the ESXi):

esxupdate --bundle=/vmfs/volumes/d3a29a6f-aa4be900/SVE-be2net-2.102.518.0-offline_bundle-329992.zip update

Once the server rebooted, we didn’t see dropped packets anymore, even under heavy load!

So we can never overemphasize the need of testing, testing, and always testing before starting production load (all components need to be stressed, CPU, memory, hard drives, etc…)! And having a good knowledge of your infrastructure is mandatory in order to know what’s going on (again thanks to the “croissants-baguette” alarm from Super Hypervisor !)

Aside, just for the fun, here is a little teaser of the unfinished plugin:

The issue we had started as we were running the script that get the *.vmx and *.vmdk files. The script started to get the files just well, but at some point (as we have a lot of VM) some errors showed up like this one:

In the same time, we had other problems, like vCenter MOB being unavailable or being unable to get VM console:

After a quick search in VMware KB, we found the KB 2033822 : vCenter Server returns “503 Service Unavailable” errors

Here is an excerpt of the KB that explain what was happening:

The vpxd log files contain entries that indicate that a socket connection attempt failed because it timed out. If you run netstat -an on the vCenter Server host machine immediately after the error, you will see many connections where one end is port 8085 on the loopback and the other end is another port on the loopback. Some of these connections will be in the TIME_WAIT state.

vCenter Server uses TCP connections on the loopback (localhost) for Remote Procedure Calls (RPC) to dispatch client requests and to communicate with vCenter Server companion services. As a result, under heavy loads, vCenter Server creates many local TCP connections, then closes them and opens new ones. Some of the closed connections remain open at the server side in theTIME_WAIT state for some time (four minutes with default Windows settings). Because the number of client-side ports is limited, if vCenter Server uses the connections fast enough, at some point the client side tries to reuse a port while the server side still has a connection for this client port in the TIME_WAIT state.

As we checked the vCenter server for TIME_WAIT connections on 8085th port (with the command netstat -an | findstr “8085.*TIME_WAIT”), we saw that there was a lot of them:

The KB 1030246 : Port 8085 in VMware vCenter Server give some explanation about port 8085 in vCenter:

This means that 8085 is the port where all the SDK connections to vCenter Server are being made, which in turn means that vSphere Client and any scripts built on vCenter Server SDK use this port.

And we can find the same alert as in the previous KB about heavy load:

If there are a lot of scripts or applications making connections with vCenter Server, it is possible to see a large number of ports in a TIME_WAIT state. This is normal because Windows keeps a socket in TIME_WAIT state for certain period of time (Twice Maximum Segment Lifetime, so this wait could be 4 minutes) before recycling it back for use.

As we have around 2000VM in this vCenter, and as we automate everything we can, we  had reach the limit for SDK connections ports. The KB gives the workaround in order to change that limit:

By default, vCenter Server has 3976 ephemeral ports. If you are running out, you can increase the limit.

To allow more local ports to be available:

• Open Registry Editor (Regedt32.exe).
• Locate this key in the registry:
• HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\Tcpip\Parameters
• Right-click Parameters and choose New DWORD Value.
• Enter MaxUserPort in the Name data box, enter 65534 (Decimal) in the Value data box, then click OK.
• Note: The default setting for the MaxUserPort value is 5000 (Decimal).The maximum value for Windows Server 2003 is 65534 (Decimal).
• Close Registry Editor.
• Restart the machine for the new setting to take effect.

If you want to fix this HTTP503 error without changing this value, you can wait for the 4 minutes delay in order to let Windows cleaning up the TIME_WAIT remaining connections, or you can restart the Windows service VMware VirtualCenter Management Webservices (for the most hurried ones).

Finally, we were able to change this settings in order to keep using massive PowerCLI scripts OneLiner ^^

This is the second VMworld we assist to and this time we’d the chance of going with our pal hypervisor.fr and to meet Timo there !

This year was radically different from the last one. Maybe because we were not alone this time, or because it wasn’t the first anymore :p In any case, we really enjoyed a lot !

Thanks to the last year experience, we’re able to better schedule this one, and to better focus on meetings than sessions. What we wanted to say, is that unless you had some questions to ask during a session, you can catch them up with Online Sessions Videos. Of course it’s not the same thing as living it, but there are so many things to see/do, you have to prioritize :p At the end, there were so many people we wanted to meet we switch on “Where is waldo?” mode :p

We wanted to focus on Group Discussion and Meet The Expert sessions, basically becauses theses are more “intimate” and everything is based on attendees participation (the Meet The Expert session was an awesome 2:1 chat with Duncan :p ).

One of the nice session we attended to was the one directed by Kit Colbert on Understanding Virtualized Memory Performance Management (INF-VSP1729), we took some pictures:

Then, there was the traditionnal Alan and Luc “Must-See” PowerCLI Best Practices: The Return! (INF-VSP1329) :

We’re also able to assist the #NotSupported #BrownBag sesssion by William :

We didn’t miss the occasion to take some pictures with the PowerCLI Gods Alan and Luc :

Finally we had the chance to chat with a lot of awesome guys: Luc, Alan, Duncan, Cormak, William. We didn’t had the chance to see Franck, we’ll try to catch up on the next belgium VMUG :p