Using vSphere and HW offload for Improved Celerra VSA performance

UPDATE June 6th, 11:34pm – with an even easier way to do it with no editing of vmx files.. (read for details)–

You’ve got to love the collaboration the Internet enables!   So – bit of background – the Celerra VSA is relatively heavyweight – needing a fair amount of RAM, and even in large memory configurations, the GUI performance is considerably slower than a real physical Celerra – it’s a very useful (and freely available here) tool – but the slow GUI can be frustrating.

So – on the EMC support forum there was a dialog about how to optimize this (thread is here: http://forums.emc.com/forums/thread.jspa?forumID=191&threadID=116336)

One of our EMCers found a little tweaking to force hardware offload of certain hypervisor elements made a massive (think 10x) improvement.

If you want to understand how to do this on VMware Workstation 6.5 and vSphere, read on….

Specifically, adding the following lines to a host running VMware Workstation 6.5 with a CPU with hardware offload (confirmed with Intel VT enabled in the BIOS – but expected to work with AMD-V also) does the trick:

monitor.virtual_mmu = "software"
monitor.virtual_exec = "hardware"

It’s now also been tested with vSphere 4 – a big shout-out and my thanks to Bernie Baker – one of our EMC VMware Specialists, and Matt Proud – the original poster to the EMC Forum.

The observed performance gains are relative to GUI performance and boot time (2 minutes and 10 seconds).  The GUI web services still take   approx. 6-10 mins to finish loading. Screen flips in many cases were sub 3 second relative to 30 seconds without the change.

The changes outlined below require a VT (Intel) enabled processor.      VT must be enabled in the BIOS and the modifications to the .vmx file have to take place an ESX host with access to the datastore on which the VSA resides.

UPDATE June 6th, 11:34pm – there’s an easier way to do this without text editing the vmx file…   Selecting “Use Intel VT-x/AMD-V for instruction set virtualization and software for MMU virtualization” in the Virtual Machine properties/options dialog does the same thing..

Bernie tested on an Intel E5345 (Single Socket Quad Core) 2.33 GHz with 8GB RAM (two servers one has 8GB one has 4GB) - Fujitsu Siemens, Primergy RX200.   I repro’ed the same results on my Intel-based Q6600 whitebox using the Intel G33 and P35 motherboards.

It hasn’t been tested on a similarly equipped AMD platform.  If somebody could/would that would be great!

VM Settings:

Performance Snapshot: (idle) – normally this would have consistently been MUCH higher.

Another proof point: 

Six simultaneous “Clone from Template” sessions.  6GB each to two iSCSI attached VSA LUNs.  Without VT and .vmx file changes this would have been pegged at 100%....

Would be VERY interesting if someone has access to a Nehalem system to see the effect of enabling the hardware MMU offload (set monitor.virtual_mmu = "hardware”) also… I’m going to try to see if I can play myself – but if somone wants to beat me to the punch, would love to see the result!

HAVE FUN!!!!

Comments

[Andrew Storrs]

Fantastic news Chad! I'm off to tune things up. :)

[Itzik Reich]

just wanted to share my experience with installing vSphere in VMware workstation 6.5.2 and what i was / wasn't able to achieve:

hardware ingidients:
1. intel i7 motherboard.
2. 12 GB of RAM
3. Windows 7 64bit as the base OS
4. VMware workstation 6.5.2

two esx4 servers installed, configured with:
esx's VM's running under workstation 6.5.2 to automatic mode
(they will pick the latest intel-vt-x)

Use "Other 64-bit" as the guest OS type, and manually add the
following to the configuration file for each ESX VM:
monitor_control.restrict_backdoor = TRUE
(this is extremley important as it will allow nested VM's to run under the Nested ESX's)

If you are using ESX Classic, you should allocate a minimum of 2 vCPUs
for an ESX 4.0 VM. For ESXi, you can get by with one.

set the nested VM (windows xp) cpu/mmu virtualization settings
at 'automatic' if you like.
edit the nested VM's vmx file to have:
replay.allowBTOnly = TRUE
(without this line, FT will NOT work..)

that's it..couldn't test powerpath as i didn't have the licenses, but i could multi-path my iscsi connections (RR) using duncan article:

http://www.yellow-bricks.com/2009/03/18/iscsi-multipathing-with-esxcliexploring-the-next-version-of-esx/

also, SRM does not work, nothing to do with the vmware workstation..vSphere is not working with SRM.

Itzik Reich

Solutions Architect

VCP,VTSP4,MCTS,MCITP,MCSE,CCA,CCNA

EMC²

where Information Lives

[Chad Sakac]

And folks - THAT is why Itzik here is one of the best of our army of VMware Champions here within EMC :-)

Itzik - you rock!

BTW - the current release of SRM(and VMware View) don't work with vSphere. They are both coming relatively shortly. In

fact the beta for the next release of SRM is starting soon, and already is dominated by EMC customers :-)

[David Barker]

Well, I've just tried monitor.virtual_mmu = "hardware" on a shiny new dell m710 - it works, but doesn't seem to add or subtract from the vsa's performance.

Still, all is nice and quick :-)

[Erik Bussink]

After having downloaded the latest Celerra Simulator 2DM 5.6.43.18 from Powerlink, I realized that I needed to also add the following line to the .vmx configuration

monitor_control.restrict_backdoor = "true"

to get passed the Blackbird slow startup.

[Mark Soho]

I have a VSA on both my laptop and on one of our esx servers (3.5). The two config options seem to help with the workstation instance, but the esx instance still idles around 25% and hits 90-100% cpu when going through the web gui. Should I just go ahead and wait till the upgrade to vSphere? Thanks!

[Chad Sakac]

@Mark - the config changes only matter in ESX 4 (vSphere ESX 4) - not ESX 3.5, so to get the same effect you see on Workstation 6.5 (hardware offload enabled by Intel VT and AMD-V)

[Dani]

Will it work on ESXi4?

[Itzik Reich]

Dani, yes it will

[kevin]

Will this configuration work with any Intel i7 motherboard? Itzik, what model processor are you using? I just want to make sure what I purchase will work with vSphere. Thanks!!!

[DC]

Hi Chad,
I asked this in the emc forums but haven't had a huge response. So if you dont mind i thought i would ask through the comments of the blog post.
I was wondering do you know if the celerra VSA supports spc-3 Persistent Reservations? I'm trying to get it to work with hyper-v.

cheers
DC

[Chad Sakac]

@DC - the Celerra has supported SCSI-3 compliant (works with W2K8) persistent reservations on the iSCSI target since DART 5.6.39. You're good to go!

[Ben Conrad]

Great videos! I've got VT-x enabled and the GUI performance is snappy, host CPU is relatively low.

[NOTE] I'm not a Powerlink subscriber, and my EMC forums (ENC) account is showing that I don't have access to some content.

I'm having issues with the write performance on my LUNs. When I've got an iSCSI LUN mounted I can only write with 'dd' at about 1.3MB/s. If I dd in /tmp while logged into the Celerra locally (/dev/sda2; EXT3) I get about 55MB/s. Same symptoms using 2 different vSwitches. I'm using the VMWare LSI Logic parallel SCSI adapter that came with the simulator.

It's as if the LUNs have no write-back cache?

I don't care about data integrity, I just need 10-20MB/s of sequential write I/O to do some basic VM testing.

Any thoughts?

Ben

[Ben Conrad]

At Storage > Systems > Clariion CX600 it says 'Write Cache Size: <1MB'.

Ouch, is there a way to get it to 64-128MB?

Ben

[Pete Karelis]

Ben,

I've created a virtual-hardware 7 version of the Celerra VSA using VMXNET3 and PVSCSI. Works quite well so far. If interested, the download is available here:

http://www.synfulgeek.com/main/index.php/articles/68-emc-celerra-vsa-great-tool-now-lets-break-the-shackles

[CC]

Chad,

I have an opteron box at home that I can use. I'll bench it and try to get the proper settings for CPU based MMU. I'll let you know what I can come up with.

[William]

I use i7 860 8GB WS 7.1 on XP x64. 2 ESX4 servers work fine. Nearly all new CPUs meet the vSphere requirements.