Virtual Geek

Using VMware? Want more perf? Help me help you.

So, everyone wants more performance, always.  

As I mentioned in yesterday’s post, it’s REALLY hard to generate a “VMware IO workload”, and most benchmarks out there aren’t that good at the IO part of a VMware benchmark.

Even VMmark 2.0 which is a great CPU/memory tile-based, mixed workload benchmark doesn’t really stress the IO subsystems particularly hard.

This means that optimizing IO performance around that workload is tough.   And, storage subsystem performance is always very workload-centric.

For example – we demonstrated face-melting, record-setting NFS performance in the SPEC SFS benchmark that I pointed out here.   So surely we have ROCKING NFS performance for NFS datastores on VMware on that same config, right?

Not necessarily.   VMware NFS datastores (and Oracle on NFS) have a very different IO profile from a general purpose NAS use cases.  They are much more transactional, so have a much higher random write mix, with vSCSIstats generally showing skews towards small IO sizes than you would see with “normal” NFS use cases that SPEC SFS simulates.

While VNX (and the previous EMC NAS and Unified generation) has generally decent VMware NFS datastore performance, but we know it can be even better.   Engineering has made a series of optimizations that seem to have have a very material effect on VMware NFS datastore performance, but the best way is to check that against workloads that LOOK like customers (and check for regression in other workloads).  We need to do that BEFORE we make it a maintenance release (which needs to go through a testing cycle), and check it into the next major release (which will of course go though a beta cycle).  For that internal test, customer workload profiles are GOLDEN.

Beyond that we’re constantly working on VMware-focused features and use cases – for both VMFS and NFS use cases – and for that, customer workload profiles are GOLDEN.

BUT – believe it or not, we don’t have a good grasp on a broad set of “normal” customer workload characterizations in engineering proper (beyond support cases where something is by definition wonky).

So – we’re going to try to use stuff Clint Kitson, one of our vSpecialist rockstars (Clint, you are a star, I love you!) developed on a series of customer engagements to help not only EMC, but the community as a whole, and make it simple and rewarding for you.

This is the “Powershell toolkit” for storage IO profiling we’re using all over the place now.  Useful in many contexts.

• Powershell scripts for collecting, processing, and federating ESXtop data  (useful for any VMware customer)
  • https://community.emc.com/message/530821
• Powershell scripts for collecting and processing vSCSIstats  (useful for any VMware customer)
  • https://community.emc.com/message/527489#527489
• Powershell scripts for EMC performance statistics block/NAS (useful for any EMC customer)
  • https://community.emc.com/message/526152#526152

Here’s the FIRST ask: If you’re a vSphere customer (of any shape/size, and EMC or non-EMC storage), please use these tools to gather a profile of the storage workload in your environment and share it on the Everything VMware @ EMC community (please zip the files, they are highly compressible).  

I will provide an iomega IX2 to any customer who shares the details on their environment, including this raw data.   The data will be public, which means it can be used by EMC and non EMC folks – and that means we’ll be able to better test, innovate, refine around the VMware use case, perhaps even built a VMware IO-centric benchmark.

Please post your environment info and the output from the scripts on this thread.

Here’s the SECOND ask: If you are a customer using EMC Celerra, EMC Unified, or VNX storage with NFS datastores, and have the ability to test an optimization in a non-production (but in use) NFS datastore environment, please let us know.   If you are willing/able to do this please comment on this virtualgeek post, I’m sure I can find some additional iomega IX2s for anyone who provides feedback on the results they see from this optimization, I’m pretty pumped about the data I’ve seen.

“…A watershed benchmark moment”

Well – it’s on record that I don’t always agree with the superlatives/dramatic wording used by the Reg.    I still think that, but it is nice that yesterday’s was in EMC’s favor :-) 

I guess that’s the lesson for me.  I shouldn’t be as sensitive.   “Superlative” words are often “in the eye of the beholder”.   If I put myself into other people’s shoes – I bet I use phraseology that might drive others nuts – without even knowing it.

Here’s Chris’ take on the record-breaking SPEC SFS (an NFS benchmark).

When we did the mega launch in January, we broke records, yes, but we also talked about 2011 being a year of EMC breaking many records.   This is only the beginning of a crazy year.  

This blog discussed the business side of things, but also (after the break) the “behind the scenes” technical part of this topic…

So…  What’s changed on the topic of public benchmarks at EMC?   I’m very lucky (and very conscious of that) to have a front row seat.  With that perspective (and also in a small way as an active participant in the debate), you can absolutely say that we’ve undergone a change – and change that a lot comes down to individuals. 

Our platform performance has been good (not always perfect), and very defendable on the competitive landscape.  

At some point in EMC’s past (before me joining so I can’t say personally) something seemed to happen that seemed to have an effect of there being a companywide informal (but pervasive) view of “performance data is complex and can be easily misinterpreted – so make it internal, and we don’t participate in external benchmarks”.   Yeah, there were some exceptions (ESRP being one as an example), but as a general principle that world-view held.

But – since a lot of recent changes at EMC over the last few years, the view of “don’t benchmark/benchmarket” or “don’t descend to that tit/tat” which used to hold sway is now a world view that would conflict with a lot of our executive team. 

Personally, I never understood the older world-view.  Like it or not, benchmarks/benchmaketing clearly matters – though I agree with the view that benchmarks are very easily twisted or misinterpreted.

Net: you can expect more public benchmarks from EMC of all kinds.  I expect it’s going to be a veritable flood.  

We’re going to do them in all sorts of forms – some that are “Face-melting performance.  Period.” (this was one of those).  Some will be “Price/Performance”.  Others will be “Performance with tons of features in funky use cases”. 

And yes, you can expect every benchmark to come with a flood of competitors responding that for one reason or another it’s not a valid benchmark.   BTW – when we weren’t doing the benchmarks, we would say something similar, so I understand where they are coming from.

With all the inevitable back-n-forth, I think it is important for customers to realize that benchmarks are by definition artificial to varying degrees.   There is, IMO a “continuum of value” in benchmarks – a reflection of how close that benchmark is to a real-world use case.  

In some cases where “single scenario” benchmarks (like SPEC SFS) really have significantly less relevance.   One example would be the very large scale, very mixed workloads we see at the large scale enterprise array customer use cases.  For those there is really a demand for a new type of storage benchmark.   These can be VERY relevant if that’s the only workload you put against the array – but more often than not, customer blend workloads.   At best in that case, the “single workload” benchmarks can hint at architectural goodness/badness (but doesn’t correlate 1:1).

More on this topic in a blog post tomorrow.  For example even with a “NFS workload” there are differing types – for example, VMware on NFS worklods.  SPEC SFS is a good indicator of more traditional NAS workloads.

The ideal benchmark IMO would be kind of “VMmark-like” in the sense that it would need to be a composite of many “real world” workloads, and would need to be able to scale to very large numbers of hosts, all very random.   Those kinds of workloads are very tough to simulate (look at how hard it is to do a really, really good VDI workload).

All that said – I’ll reiterate… you CAN expect lots more on the public benchmarking and benchmarketing from EMC – feedback on whether it’s good or not is welcome.

What’s the nutshell on this one?

• The EMC VNX performed at 497,623 SPECsfs2008_nfs.v3 operations per second – a new SPEC SFS record.
• That result is significantly faster than the previous leaders: 49% faster than an HP system (BL860c i2 4-node HA-NFS Cluster), 23% faster than an IBM system (IBM Scale Out Network Attached Storage 16-node cluster, Version 1.2), and 160% faster than the NetApp FAS 6240 Series system.
• The EMC VNX Series also had the lowest overall response time of systems tested, taking the top spot with a response time of .96 milliseconds. 
• The major factor in this result was the fact that we’ve been very focused on being able to leverage (not just use, but scale to squeeze the most out of) the very large multi-core performance of Intel’s Westmere tech, and couple that with very pervasive use of SSD (as both cache and a tier).   The amount of R&D focused in VNX-land on how we scale up by adding more blades that handle NAS or block duties, how each of those scales up as more cores are added per blade is as much of an engineering effort as the parallel efforts in VNXe-land around being able to run all those functions on one kernel and one blade.

If you want to see EMC’s result (and everyone else’s) click on the link below.

While “world record breaker” is the headline (and that’s still only using 4 active datamovers where we could go to 8, or 7 with a standby, and could have added a lot more SSD and westmere cores on the backend if we wanted to) – there’s a very important story behind the story.

If you’re interested in the insider engineering half of the equation read on….

Why Virtualize Oracle? How about saving $5M and 20x faster perf?

While there are MANY enterprise customers who are virtualizing mission critical apps, some who are virtualizing clients in all shapes and forms, some who have functioning internal self-service ITaaS portals, and some who are leveraging external public cloud models – for the most part, we’re still in relatively early days.

Paul Maritz at PEX talked about how today, while there are many advanced examples – as a whole – most customers are just finally getting done at virtualizing the “craplications” (aka as “Phase I of the VMware journey”), and are starting to virtualize things that carry heavy SLAs.   In my experience – the existance of SLAs is the defining element of a “mission critical application” vs. a “craplication”.

FTW, obviously I’m being a little facetious when using the word “craplication”.  These apps, generally “owned and operated” by IT are very important, and virtualizing them is a no-brainer.   What they don’t have is high “risk” or SLAs associated with them.

When you ask about a given application: 1) “what are the specific performance requirements”; 2) “what are the disaster recovery and retention SLAs”; 3) “does the application or it’s data require compliance with security policies and is it audited” you get two very distinct responses.

• Craplication = “huh?   Look, we have a lot of these running on 1U/2U servers, one NIC, and local storage.  I have no idea about it’s backup, performance or security issue.”
• Mission Critical Apps = “yes, it does have those SLAs, and here they are”.

Now, for the craplications, the “huh” translates to “don’t worry.  Let’s just virtualize - it full steam ahead.  We’ll deal with stuff as it comes up”.   And that process generally works really well – and has resulted in massive TCO over the last few years, and that’s all been very good.

For mission critical apps – unless you can answer with confidence that you can not only meet those SLAs, but improve them, they tend to stay “stuck” on older, less efficient stuff – because risk of change is pronounced (or at least perceived that way).

I think that 2011 will be a year of a LOT of progress on this front.

For most enterprises, there’s not any application that is more mission critical than their core business apps, which generally means either Oracle or SAP.   While it’s been a while in coming, there’s been a lot of thawing around Oracle and VMware on many fronts.  I talked about that over the last year, and you can see the latest here.

Within EMC, there’s an interesting forcing function – Joe Tucci and Paul Maritz are emphatic that both IT organizations must be ahead of our own stuff – acting as a early alpha/beta not only of the technology, but the whole idea.   Every quarter, we have a quarterly business review, and one of the many topics is what our respective IT orgs are doing to push the envelope.

We’re currently pretty far along, with 75% of our workloads running as VMs, including the vast majority of our mission critical apps, and a broadly rolled out View 4.5 deployment.  It’s still early days of self-service portals but we are using various flavors of ITaaS.

One thing that is VERY cool is that this is all being done very publicly, and with LOTS of detail (you can get details on all the EMC IT projects at www.emc.com/emcIT  

A killer example was one I mentioned at PEX – the results of replatforming our Oracle 11i/10g RAC deployment from Solaris to Linux and from Sparc to x86, and for the 11i app tier – virtualizing it on vSphere 4.1.

For perspective – this is one of the largest Oracle 11i Apps deployments in the world.   It’s also the beating heart of EMC’s business – supporting our quoting, inventory, huge parts of the CRM system and more.

So – what was the result?  

• 10-20x performance improvements.
• $5M cost savings.
• 90% greater productivity

If that doesn’t get you moving, I think you might have a serious issue.

If you’re one of the millions of enterprises with rusting old legacy big-endian systems (Power, Sparc, etc)  and considering the next evolution of the critical heart of your most mission critical systems – you owe it to yourself to read this whitepaper.   Fluffy marketing it ain’t.

ADDED Feb 24th - NOTE: I’m not saying that by definition big-endian is BAD, and little-endian is good.   What I am saying is that the speed of refresh, and commoditization forces that have applied more strongly in little-endian land mean that a unit of compute power for a unit cost seems to have swayed quite firmly to x86 (little-endian) land.   You simply DON’T see the Google/Facebook/Rackspace/Terremark (or most enterprises) building a LOT of new large scale big-endian deployments.   On this one, the relative market growth of little-endian relative to Itanium, Power, Sparc (which still are used in a TON of places that are very important) speaks more volumes than I every could.  But you see almost every enterprise having a lot of really old big-endian systems out there rusting.  That was certainly the case here. 

As an example, we describe critical lessons learnt through the process.  In some cases, it was literally so much faster, that it had 2nd and 3rd order effects that we didn’t consider in the initial process.   We found important VMware, EMC, Cisco bugs through the process.

So – what’s next?   Beyond building one of the largest and most energy efficient (BTW, 100% virtualized) datacenters in the world (a crazy cool story on it’s own), we’ve been working hand in hand with VMware on “future vSphere releases”.  One of the main goals of that next release is to handily virtualize the DB tier (currently still running on the UCS hardware as physicals).   Stay tuned – as cool as the current stuff is (really, hats of to Sanjay’s crew), we’re on an awesome journey…

Would love to hear YOUR “Virtualizing Mission Critical Apps” story….

VMware Partner Exchange 2011–Recap and Content

UPDATED Feb 23/2001 8:51am – Fixed the PEX deck link.

Well – another VMware PEX has come and gone, and PEX 2011 was a doozy   This is actually one of my favorite events of the year – it’s more intimate in some ways than VMworld.   Plus, you see people who are in an important sense - all in the boat with you – rowing as furiously as you are.

From an EMC standpoint We dialed it up to 12 this year…

• EMC was the most attended pre-conference boot camp (ones were held by Cisco, NetApp, Dell and HP).  We had # attendees, more than 20% more than the next closest – that’s the extra 20% they get for choosing EMC :-)   We must be doing something right!   This was recorded, and is below.
• We held a VMware/EMC HoL  - I didn’t see anything like it from anyone else.   Similar to VMworld, where there is a massive HoL (which like VMworld, we help do our part to power with EMC unified kit… Currently several NS-960s with SSD, 10GbE and FAST/FAST Cache)   BUT – it’s 100% VMware centric.   No opportunity (understandably) to play with VMware/EMC integrations like the VSI vCenter plugin, PP/VE, Replication Manager, Avamar and more.    So – we run a second much smaller, but no less cool HoL focused on VMware+EMC.   BTW, I talked to the VMware HoL (the big one) team who are awesome folks – would like to try it on a VNX/big Isilon cluster at future things…
• We challenged attendees to take the “EMC No BS Challenge”.  If they went to the VMware/EMC HoL and tried the stuff for themselves, then wore the “EMC No BS Challenge” hat, they won one of 3 iPads.
• We had a killer booth, with lots of schwag.   I think though that this one took the cake – one of the folks that visited our booth won a VNXe!
• There was a great VCE booth – which was packed constantly, and had a working Vblock 1 config humming along.
• We held a VMware/EMC Partner Appreciation event to thank our partners.
• We opened the exhibitor pavillion with a demo that I personally thought was, as advertised (this was the title of the session): "Face Melting VMware/EMC coolness - today and tomorrow".   This was recorded, so if you want to see for yourself, see below.

I want to thank everyone who made this happen – but most of all, Jessica Doyle.   Jessica – it wouldn’t have happened without you!   Second of all – the awesome vSpecialist Tech Enablement team (Chris, Tee, Simon)

Most important of course though was meeting friends, colleagues from VMware and the partner community.  THANK YOU!

For more detail - including recordings of the EMC PEX keynote with a bunch of crazy stuff, softcopies of the EMC bootcamp keynote, and also that face-melting demo we did in the exhibit hall… read on!

Updated 2011 vSphere ESXi 4.1 (and future) Homebrew/Whitebox parts list….

UPDATED Feb 22nd 2011, 9:40am – note that this build uses a MB that has a Realtek NIC (or more accurately MAC/PHY).   In my build notes, I purchased/repurposed Intel Pro GT/PT NICs from older whiteboxes.  If vSphere boots, but can’t load any network drivers, it will report an error like this: “vmkctl.HostCtlException Unable to load module /usr/lib/vmware/vkmod/vmfs3: Failure”.   There are two easy solutions: 1) customize your vSphere install ISO to support the Realtek NIC (great instructions here); 2) just buy the Intel NICs.   I **personally** recommend option 2 – not because option 1 doesn’t work (it does, I tried it), but because it means you have one more strange thing on an already Frankenstein whitebox :-)

So, as I’ve said before, when things get political, and get all messy – I like going into the lab and spending a solid day just playing with tech.   It’s how I get back into a zen state.   That home lab powers a ton of VMs, and let’s me play, learn and stay fresh – not just on EMC stuff (using Virtual Appliances) and VMware itself, but a lot of folks across the industry.   The home lab has 5 “mainstream” ESX hosts – and some other loosey-goosey ones I use for various purposes.

Staying technical…  Ooommm.

So – I’ve been playing with the latest ESXi 4.1 bits, and also with “future VMware software”, and noticed that my old whiteboxes – one of the most popular blogs I’ve done had the “parts list” – (basically Intel Q6600, Intel Q/P35 chipsets, 8GB of DDR2 mem, boot from USB,Intel PCI PRO/1000 GT and PCIe PT NICs) where starting to have some problems.

Specifically, with future VMware software, there were various UEFI chipset things – but I can’t blame VMware, we’re talking about very old, very non HCL hardware.

There are other great “building a home lab” articles on the internet.   I like this one, this one, and this one.

So – I did a refresh – and was stunned by how much you can get for how little.  With a disclaimer that I can’t vouch for anything other than my current experience if you want the shopping list, and the results – read on….

You can’t trust what you read on the Internet blindly.

That title is particularly true, when it comes to what one vendor says about another – I would consider it HIGHLY suspect.   Apply that to anything I say.   That’s why I try my darnest to NOT write blogs that target competitors, but try to focus on the nuts and bolts that are vendor neutral, and content that is useful to VMware/EMC customers and partners.  Sometimes I fail.   When I do – apply that filter.

Read this post through that filter – which is to say DON’T READ IT.

I’ve said it before, and I bet I’ll say it again – this is what I think:

• Trust your own personal experiences with technology and vendors most of all.
• Next, trust the experiences from friends/colleagues (who are not vendors) you trust.
• Moving further down, be skeptical, but listen to what vendors say to you about their own stuff.
• And at the very, very bottom – be extremely skeptical – almost penalizing them – when a vendor talks about the other their competitor in a negative way, unprompted.

If only customers would consistently do that, the FUD flinging would slow to a crawl and stop – but there’s some deep human instinct behind this, IMO.

Update – 8:10am, 2/11/11.  Preston de Guise commented that this sounds like I’m blaming the customer.   I can see how it sounds like that (and thanks for the courteous comment Preston).  That wasn’t my intent.  As much as I try to stop “feature list selling”/”competitive compare tables”, the pressure to produce those things is through the roof.  When I don’t do it, others do. 

My comment about “deep human instinct” is that I think people use pseudo “rationalist” approaches like this to justify the decisions they made a while ago (based on trust, relationships, and their ‘gut’).   So long as customers respond positively to these “checkbox tables” (which are invariably wrong, outdated, and biased out the wazoo), vendors WILL make them.   So long as customers ask for/respond to those sort of vendor action (and they do, they absolutely do), SOMEONE will do it.

I guess it’s like the illicit drug business.   On one level, it’s wrong to blame the users, as they are the victim, and the pusher/dealer is the criminal.  On another level, ONLY the users can actually control what happens – because as so long as there is demand, as soon as one pusher/dealer is incarcerated, another will pop up in their place.

As an interesting note – that’s why within EMC, one of the most important metrics for all our business units is something called a “net promoter score” (which asks a single question: “would you recommend this?”) – since the answer is measure of the network effect of the first 2 bullets.

If going to stick in the blog split here to spare people who are above this fray (sigh, I wish I didn’t have to do this, but what some folks continue to demand it).  

My suggestion, DON’T read on unless you are willing to listen to the original comments stated in the webcast, not the summary on The Register and then the subsequent vendor bash summary.   the actual original source recording is an incredible insight into EMC (all the way through Goulden, who was particularly interesting)…. Read on if interested (and you really, really shouldn’t be)…

Feb VMware/EMC webcasts are doozies…

Feb is a solid calendar of webcasts.   As always, I try to reinforce, these are not for marketing, but for solid technical content, from solid technical folks.  You come to a webcast to get something out of it – not to watch a commercial, and we know your time is valuable.

Same bat time, same bat channel – every Thursday starting at 8am PT / 11am ET.

1. The first one (Feb 3rd) will be on how to download, install, and use the Celerra VSA.  AFAIK, this remains the only virtual storage appliance which: a) is freely available to ANYONE; b) never times out; c) supports block and NAS storage; d) you can add as much storage as you want, e) is functionally equivalent (in every way) to a leading platform (Storage Magazine’s January 2011 WINNER in the Enterprise NAS Quality Award).   The presenter is Simon Seagrave, aka @kiwi_si – a vSpecialist out of the UK, and author of the www.techhead.co.uk blog.  He’s written a series of blog posts on using the Celerra VSA here.
2. The second one (Feb 10th) will be on how to install and use the EMC vCenter plugins.  These are all provided for free, and we’re now on our fourth gen – so have a lot of refinement (and still tons of coolness up our sleeve for the 5th gen which we’re working on now).  The presenter is Rick Scherer – a vSpecialists out of SoCal, and author of the http://vmwaretips.com blog.   He the man of a thousand accreditations, and one of the teams VCDXes – incredible talent.   Ask him ANYTHING.
3. The third one (Feb 17th) is on a great topic – VMware View 4.5, and how to get the most out of it, for the least.   There’s a ton of work that’s been done between VMware and EMC to validate a series of reference architectures, at a series of different client profiles and different scales (hundreds-thousands).   The discussion will show how to leverage everything VMware and EMC does to get hard-dollar cost per client to around the $500/client mark.   The presenters are Brian Johnson and Josh Mello – two of the key members on the EMC’s proven solutions team.
4. The fourth one (Feb 24th) is on securing the VCE Block and Virtual datacenter.   This is a crazy hot topic, and there’s some VERY, VERY cool tech behind it.   Put it this way – in Q4, a customer refused to buy a Vblock (or any converged infrastructure for that matter) until the CISO saw this stuff, at which point they signed a $8M PO.  On the spot.   The presenter is Mike Foley – again, a killer vSpecialist, one of the folks on the team that focuses on security.   He’s been working on virtual appliances and PowerShell for security and vSphere purposes – if you want to ping him on stuff you’re curious about.

Click on the link below to register for one or all of them: