vSphere 4 NMP RR IOoperationsLimit bug and workaround

I mentioned that we had found an issue in my vSphere 4 update 1 post (not the same one related to APD path state discussed here – please read that post – seems to be affecting a fair number of customers), and there’s now been enough research to validate the bug, so I want to explain it in more detail.

So, here’s the story.  Our Cork Solutions Center team (an awesome crew!) was doing some testing with CLARiiON and vSphere 4 with Exchange 2007.   The Jetstress runs showed weird inconsistent results, that looked like this:

 

This is a Navisphere Analyzer (a CLARiiON performance tool) showing the IOps per front end interface at different user loads.   Note how all the front end ports are nicely loaded in the 20K, 16K, 12K, 8K, 4K user workloads, but then are all messy for the 24K user run?  What happened?   Well – the ESX host was rebooted between the 4K run and the 24K user run.

A bit more background – we try to do as much testing with both vSphere 4’s native multipathing (NMP) using round robin (RR) and also the EMC PowerPath/VE vSphere vmkernel loadable multipathing plugin (some customers will use both).   In the NMP RR tests, we switched the IO Operation Limit parameter to 1.   This parameter controls how many IOs are sent down a given path before vSphere starts to use the next path.   By default, this value is 1000.

This is what the default shows if you issue a command from the vMA (BTW – the “Use Active Unoptimized Paths” value makes RR use ALUA paths that advertise a “unoptimized” state – meaning they are on a “non owning” storage processor)

After running “esxcli nmp roundrobin setconfig --device=naa.<the device naa goes here> --iops 1 --type iops” for each device, it looks like this …

But we noticed after a reboot – the IOOperationLimit value reverted to a weird random value (in this case 1501691480)

This is what resulted in the weird behavior. 

Since, this has been confirmed to also be the case in vSphere 4 update 1.    So – until this is resolved, just leave the IOOperationsLimit at the default if using NMP Round Robin.   Note that this doesn’t affect customers using PowerPath/VE, where the number of I/Os issued down one path/queue is adaptive based on queue depth (when using HDS or HP arrays) and is predictive (adaptive, but also using target port queue depth as a predictor of future initiator queue depth in the algorithm) when using EMC arrays.

Comments

[Gunnar Anderson]

Even if you have the default value of 1000 which you get when you change to RR and don't change the IOPS value to 1 the value will be changed when you reboot the server. So our solution for the time being is to set iops = 1 after each reboot.

[stuart]

The solution is edit the rc.local file (which basically is a script that runs at the end of boot up) to set the limit IOPS limit on on all luns

This is in the /etc

Run a nano and Enter this command, the only variable you will need to change is the “naa.600” witch pertains to the identifier on array

for i in `ls /vmfs/devices/disks/ | grep naa.600` ; do esxcli nmp

[stuart]

The solution is edit the rc.local file (which basically is a script that runs at the end of boot up) to set the limit IOPS limit on on all luns

This is in the /etc

Enter this command, the only variable you will need to change is the “naa.600” witch pertains to the identifier on array

for i in `ls /vmfs/devices/disks/ | grep naa.600` ; do esxcli nmp roundrobin setconfig --type “iops” --iops=1--device $i; done

http://virtualy-anything.blogspot.com/2009/12/how-to-configure-vsphere-mpio-for-iscsi.html

[cheap viagra]

I totally agree with this comment, thanks for sharing, have a nice day!!

[Koen]

Any idea how this can be done on an ESX4i update 1 ??

[chouse]

Koen: this can be done just the same as on ESX. You just need to enable the unsupported Tech Support console and add the script posted by stuart to /etc/rc.local.

[C Skovdal]

Hi all
I found a workaround for the reset of iops values after reboot.
just run use the --config. And the changes persist after reboot.
Got a heads up from vmware that the error bug should be fixed in update 2

for i in `ls /vmfs/devices/disks/ | grep naa.600|grep -v :1` ; do esxcli nmp psp setconfig --device $i --config "policy=iops;iops=1;useANO=1"; done

Kind regards,
Christian Skovdal - Denmark

[Ben Conrad]

Looks like this is fixed in 4.0 U2, I have not tested it yet:

http://www.vmware.com/support/vsphere4/doc/vsp_esx40_u2_rel_notes.html

For devices using the roundrobin PSP the value configured for the --iops option changes after ESX host reboot
If a device that is controlled by the roundrobin PSP is configured to use the --iops option, the value set for the --iops option is not retained if the ESX Server is rebooted.

This issue is resolved in this release

Ben