We’re gotten some small scale reports of customers who are finding that after running after some time with VAAI on their VMAX arrays are finding performance degradation. Usually this correlates with excessive back-end load and high read cache misses.
This is covered in primus article emc261696 (http://knowledgebase.emc.com).
Don’t jump to a conclusion – if you think you might be experiencing an issue – always open an SR. You can always call 1-800-SVC-4-EMC, and better yet – the e-support is getting really, really good (http://support.emc.com). Interestingly, online chat support is now covering a lot of support cases, and gets some of the best customer feedback.
A workaround (covered in the Primus article) is to disable VAAI on your hosts. You can disable VAAI across your vSphere clusters using this powershell script (thank you Nick). This requires rebooting the ESX host, but of course using vSphere’s maintenance mode, you can do this non-disruptively
Good news is there is already a fix. It is in a Enginuity epack that is available now. You get this via support.
Also good news is that the fix is in the Enginuity service release scheduled for 3/30.
Will the Enginuity epack also fix the issue in emc252367?
Posted by: Eric | March 29, 2011 at 01:08 AM
The primus article mentioned is not available for customers on powerlink. My ASR pointed me to a different one: emc263675.
Do you have any more info on the Enginuity service release scheduled for 3/30??
Posted by: Stanley Merkx | March 30, 2011 at 04:38 AM
The VAAIs can be turned on and off dynamically through the vSphere Client, even during operations that take advantage of them. There will be no disruption in an ongoing operation as it switches between either using the API or not. Ergo, no reboot of the ESX host is required.
Posted by: Drew Tonnesen | April 04, 2011 at 05:05 PM
Now, what about the ESX crash-causing bug between PowerPath and VAAI? Let's get some coverage of that issue please. It's hit us and I understand there are more EMC customers who have experienced this. The irony here is that we spent *more* to get PowerPath, and now we have to turn off VAAI because of this bug!
Posted by: Craig Dodson | April 12, 2011 at 01:13 PM
@Craig - thank you for being a customer. This issue has just popped on my radar, and the crew is digging deep.
It's too early to point to root cause, but looks like it might be a very specific condition with a given HBA (but widespread). Regardless, we are responsible for resolution to your satisfaction.
Question - do you mind emailing me your specific SR #?
Posted by: Chad Sakac | April 12, 2011 at 04:16 PM
@Eric,
The Enginuity epack will not fix the issue in emc252367.
@Craig,
EMC, VMware and Emulex are working together on this issue. It is not considered to be a PowerPath bug. At this time the cause is unknown. Our efforts are being hampered by the inability to reproduce this in house. Two customers that had seen the problem have been working with us but they have not seen the issue since.
Posted by: Keith Kearnan | April 12, 2011 at 05:01 PM