IMPORTANT: VMAX epack and vSphere hotfix

Ok – I made a post here about a couple of issues.  The first wasn’t really an EMC issue per se, but since the elab support matrix is followed to the letter, and there was a known timing issue with an Emulex firmware release, I wanted to point it out.

The second (in the span of the few days since the post) has popped up in enough places (still only a few, but it doesn’t matter – rare or not, customer pain is bad) that everyone at VMware and EMC went into furious high gear on this over the last few days.

So – an ETA (these are EMC Technical Advisories) went out on June 7th.  This goes to all EMC customers, but I wanted to draw EXTRA attention to it.

The core issue is related to VAAI and VMAX arrays running Enginuity 5875.135.91 or 5875.139.93.

Remember, since VAAI is on by default, if you have upgraded to vSphere 4.1, and your VMAX to Enginuity 5875 (those specific revs) – this applies to YOU.

The detail of the issue, and workaround are posted in EMC primus (http://knowledgebase.emc.com) article emc263675.

While I encourage customers/partners to read the ETA and the primus article here’s the key of the fix:

• Do NOT use the VAAI Copy offload function until an available Enginuity Pack is loaded. An alternative is to disable the advanced parameter DataMover. HardwareAcceleratedMove by setting the value to 0 (the default is 1).

  Note: You can disable VAAI across your vSphere clusters using this powershell script (thank you Nick).   This requires rebooting the ESX host, but of course using vSphere’s maintenance mode, you can do this non-disruptively. UPDATED 6/16 10:50am ET – a bit of a space-out moment.  Of course, changing the ESX datamover from software to hardware (changing the advanced property) doesn’t require a reboot.

• An Enginuity Pack at the current Target code, 5875.139.93, is available to prevent this issue. It should be loaded to replace 5875_135/0084 / 5875_091/5011. The new Enginuity Pack adds fixes 56516, 58269, 58328, 58546, and 58566 at 5875.139.93, R/B 5875_139/0013 and is available from the SSR GA Enginuity Pack download page. Environments at Enginuity 5875.135.91 and 5875.139.93 using VAAI should load the Enginuity Pack and the VMware patch at the earliest opportunity.
• In addition to loading this Enginuity Pack, customers must also contact VMware Support directly and request a Hot Patch. This hot fix is required to fix an issue encountered when using VAAI for Storage vMotion and Cloning operations where performance degradation can result due to the the operation(s) taking longer to complete than when using Software (Host) Data Copy.  VMware is currently working to release this fix in a GA version of ESX 4.1 targeted for release mid Q3 2011. However, if you encounter this issue, report the problem to VMware Technical Support to receive the Hot Patch.  All customers with a VMware Support contract can file a support request using the instructions on the support site: http://www.vmware.com/support/contacts/?x=42&y=13.

For what it’s worth – my apologies to customers, we should have caught this pre VAAI GA. 

There’s a lot of post-mortem that has gone on, and continues to go on, and we have changed our going forward testing and qualification model around things like this to improve. 

While it doesn’t make up for a miss, we try to hold ourselves to a high standard, and that means being blunt, owning up to the error, being honest and self-critical, and then fixing the core process.

Again – thank you to our customers for being joint VMware and EMC customers!

Comments

[Drew Tonnesen]

The VMware fix is not in the 5.0 RC build. Also please note that you do NOT have to reboot ESX/ESXi hosts to disable VAAI - it is completely dynamic.

[Dave Lawrence]

Chad, thanks for the info, just had a customer run into this.

Here's the VMware KB for reference as well:

http://kb.vmware.com/kb/2000755