So, everyone wants more performance, always.
As I mentioned in yesterday’s post, it’s REALLY hard to generate a “VMware IO workload”, and most benchmarks out there aren’t that good at the IO part of a VMware benchmark.
Even VMmark 2.0 which is a great CPU/memory tile-based, mixed workload benchmark doesn’t really stress the IO subsystems particularly hard.
This means that optimizing IO performance around that workload is tough. And, storage subsystem performance is always very workload-centric.
For example – we demonstrated face-melting, record-setting NFS performance in the SPEC SFS benchmark that I pointed out here. So surely we have ROCKING NFS performance for NFS datastores on VMware on that same config, right?
Not necessarily. VMware NFS datastores (and Oracle on NFS) have a very different IO profile from a general purpose NAS use cases. They are much more transactional, so have a much higher random write mix, with vSCSIstats generally showing skews towards small IO sizes than you would see with “normal” NFS use cases that SPEC SFS simulates.
While VNX (and the previous EMC NAS and Unified generation) has generally decent VMware NFS datastore performance, but we know it can be even better. Engineering has made a series of optimizations that seem to have have a very material effect on VMware NFS datastore performance, but the best way is to check that against workloads that LOOK like customers (and check for regression in other workloads). We need to do that BEFORE we make it a maintenance release (which needs to go through a testing cycle), and check it into the next major release (which will of course go though a beta cycle). For that internal test, customer workload profiles are GOLDEN.
Beyond that we’re constantly working on VMware-focused features and use cases – for both VMFS and NFS use cases – and for that, customer workload profiles are GOLDEN.
BUT – believe it or not, we don’t have a good grasp on a broad set of “normal” customer workload characterizations in engineering proper (beyond support cases where something is by definition wonky).
So – we’re going to try to use stuff Clint Kitson, one of our vSpecialist rockstars (Clint, you are a star, I love you!) developed on a series of customer engagements to help not only EMC, but the community as a whole, and make it simple and rewarding for you.
This is the “Powershell toolkit” for storage IO profiling we’re using all over the place now. Useful in many contexts.
- Powershell scripts for collecting, processing, and federating ESXtop data (useful for any VMware customer)
- Powershell scripts for collecting and processing vSCSIstats (useful for any VMware customer)
- Powershell scripts for EMC performance statistics block/NAS (useful for any EMC customer)
Here’s the FIRST ask: If you’re a vSphere customer (of any shape/size, and EMC or non-EMC storage), please use these tools to gather a profile of the storage workload in your environment and share it on the Everything VMware @ EMC community (please zip the files, they are highly compressible).
I will provide an iomega IX2 to any customer who shares the details on their environment, including this raw data. The data will be public, which means it can be used by EMC and non EMC folks – and that means we’ll be able to better test, innovate, refine around the VMware use case, perhaps even built a VMware IO-centric benchmark.
Please post your environment info and the output from the scripts on this thread.
Here’s the SECOND ask: If you are a customer using EMC Celerra, EMC Unified, or VNX storage with NFS datastores, and have the ability to test an optimization in a non-production (but in use) NFS datastore environment, please let us know. If you are willing/able to do this please comment on this virtualgeek post, I’m sure I can find some additional iomega IX2s for anyone who provides feedback on the results they see from this optimization, I’m pretty pumped about the data I’ve seen.
Gathering the data now. We also have an NS-480 and an NS-120 that we use for production/development that I would be happy to try out performance tweaks on.
Posted by: Matt | February 25, 2011 at 10:06 AM
Working on the PS Scripts now.
Posted by: Steve | February 25, 2011 at 11:32 AM
Will be happy to help out either way, but would you ship an IX2 to Denmark?
Posted by: Tom | February 25, 2011 at 12:01 PM
This is a fantastic idea, and I 100% agree that using real (anonymized) workloads is the best possibly way to improve performance. I know that even as an engineer working for a competitor I love being able to get real workload data from a customer to size/optimize whatever they have (and maybe replace it, eh?).
BTW, its worth mentioning that the data from vSCSIstats doesn't include your actual data sent down the pipe, so its reasonably safe and you aren't giving anything private away. However, some (including myself) might consider the internal hostnames, etc. exposed by ESXtop to be sensitive. Maybe Clint/the other vSpecialists could right another script to take the data and scrub out the hostname fields to replace them with randomized names?
Posted by: Mcowger | February 25, 2011 at 12:09 PM
I'd love to help.
Jas
Posted by: Jason Boche | February 25, 2011 at 11:08 PM
Sign me up for task #2 (optimization test on NFS).
Posted by: Jason Boche | February 25, 2011 at 11:11 PM
Count me in!
Got awesome customers who might be happy to share their storage workload profiles.
Thanks for the freebie. I hope you can deliver to Belgium :)
Posted by: Dpironet | February 26, 2011 at 12:56 PM
Count me in too! Using Celerra (ns480) nfs datastores for our dev environment. Would like to try the tweaks you mentioned.
Posted by: PJ Spagnolatti | February 28, 2011 at 08:11 AM
WE HAVE OUR FIRST WINNER!
@MillardJK is the first to post the data and I will be sending an IX2 shortly.
@Jas - Jason, the code is just about done, would LOVE your before/after feedback.
@Dpironnet - Thank you - I will absolutely ship to Belgium.
@PJ Spagnolatti - Thanks. Please post your "before" data on the forum thread as per the document!
@Matt - Thanks. Please post your "before" data on the forum thread as per the document!
@Steve - Thanks. Please post your "before" data on the forum thread as per the document!
@Tom - I will absolutely ship to Denmark, we are a global community, baby :-)
Posted by: Chad Sakac | February 28, 2011 at 10:06 AM
Count me in for the first task
Posted by: afokkema | February 28, 2011 at 10:28 AM
Just posted the data for your first ask, will run another sample next week where we should be at around 100 VM's.
For your second ask we have a second environment wit 3 ESXi's where we can do some testing.
regards
Andi
Posted by: Fox_inti | February 28, 2011 at 11:13 AM
The drive to succeed comes from the burning motivation to achieve a function. Napoleon Hill wrote, Whatsoever the imagination of dude can consider and think, the brain can achieve.
Posted by: Supra The Penny | February 28, 2011 at 07:43 PM
Hi Chad,
We are looking at away of generating a report either for all Datastores or VMDKs (aggregated across all hosts) as follows:
Read and Write IOPS, Bandwidth, Latency (Avg and Peak)
I understand from talking to one of our engineers that this not possible from vCenter or CapacityIQ.
I also see from the vSphere 4.1 Storage presentation:
"Not available in this timeframe: Aggregation at cluster level in vCenter (possible through APIs)"
Do you therefore no where we can get scripts that will use the APIs to aggregate the statics and give us a break-down of all Datastores and VMDKs?
I assume in vSphere 5 that the Storage DRS feature will use aggregated Datastore statistics and that this information will then be in vCenter.
Many thanks
Mark
Posted by: Mark Burgess | March 01, 2011 at 08:56 AM
@Mark - if you take a look at the scripts posted at the Everything VMware @ EMC community (link in this post), you'll find that they can aggregate storage performance stats across a number of hosts. We're working to add some visualization based on the raw data.
@Andi - thanks for posting the data! will send the IX2, and the update is now out - process on how to get it coming shortly.
Posted by: Chad Sakac | March 01, 2011 at 03:24 PM
Posted data yesterday for the ESXTOP stuff, the vscsi data seems to have gone wrong somewhere along the lines - running for a little longer under production load :)
Sam
Posted by: Sam McGeown | March 02, 2011 at 10:13 AM
Hi Chad
We have a mix of IX's, CX's and hopefully a soon to be delivered VNX for our lab environment. Happy to share stats and get steers from VMware on tweaks. I am trying to build seamless solutions for the branch office and the data centre with maybe IX's onsite for local storage with the option of cloud storage expansion to a DC hosted VNX or cloud backups to a DC hosted Atmos solution using just VMware and EMC products and also trying where possible to avoid client software and do everything at the hypervisr or storage layer. It's proving a challenge even though everything is single vendor. I have had a sneak peak at the new cloud lifeline interface on the IX and noticed the EC2 and Atmos icons. Iomega are giving us the SDK to see what we can do.
Posted by: Barry Weeks | March 03, 2011 at 04:07 PM
I'd be happy to help.
Posted by: Marty Kilroy | March 04, 2011 at 08:47 AM
I am not a daisy person.
Posted by: Marty Kilroy | March 04, 2011 at 08:49 AM
Just want to link the conversations ... http://virtualscoop.org/?q=blog/1
Posted by: Irfan | March 06, 2011 at 01:22 AM
Chad,
For the second task would the optimisation work on NFS datastores hosted on a Celerra NS42G with a DMX backend (rather than CX ?) ? i.e is the tweak in the Celerra?
If so I am more than happy to test.
Posted by: Rich | March 09, 2011 at 05:45 AM
Count me in for task#1. I'll post the data as soon as I can.
Posted by: Rguyler | March 21, 2011 at 04:09 PM