“…A watershed benchmark moment”

Well – it’s on record that I don’t always agree with the superlatives/dramatic wording used by the Reg.    I still think that, but it is nice that yesterday’s was in EMC’s favor :-) 

I guess that’s the lesson for me.  I shouldn’t be as sensitive.   “Superlative” words are often “in the eye of the beholder”.   If I put myself into other people’s shoes – I bet I use phraseology that might drive others nuts – without even knowing it.

Here’s Chris’ take on the record-breaking SPEC SFS (an NFS benchmark).

When we did the mega launch in January, we broke records, yes, but we also talked about 2011 being a year of EMC breaking many records.   This is only the beginning of a crazy year.  

This blog discussed the business side of things, but also (after the break) the “behind the scenes” technical part of this topic…

So…  What’s changed on the topic of public benchmarks at EMC?   I’m very lucky (and very conscious of that) to have a front row seat.  With that perspective (and also in a small way as an active participant in the debate), you can absolutely say that we’ve undergone a change – and change that a lot comes down to individuals. 

Our platform performance has been good (not always perfect), and very defendable on the competitive landscape.  

At some point in EMC’s past (before me joining so I can’t say personally) something seemed to happen that seemed to have an effect of there being a companywide informal (but pervasive) view of “performance data is complex and can be easily misinterpreted – so make it internal, and we don’t participate in external benchmarks”.   Yeah, there were some exceptions (ESRP being one as an example), but as a general principle that world-view held.

But – since a lot of recent changes at EMC over the last few years, the view of “don’t benchmark/benchmarket” or “don’t descend to that tit/tat” which used to hold sway is now a world view that would conflict with a lot of our executive team. 

Personally, I never understood the older world-view.  Like it or not, benchmarks/benchmaketing clearly matters – though I agree with the view that benchmarks are very easily twisted or misinterpreted.

Net: you can expect more public benchmarks from EMC of all kinds.  I expect it’s going to be a veritable flood.  

We’re going to do them in all sorts of forms – some that are “Face-melting performance.  Period.” (this was one of those).  Some will be “Price/Performance”.  Others will be “Performance with tons of features in funky use cases”. 

And yes, you can expect every benchmark to come with a flood of competitors responding that for one reason or another it’s not a valid benchmark.   BTW – when we weren’t doing the benchmarks, we would say something similar, so I understand where they are coming from.

With all the inevitable back-n-forth, I think it is important for customers to realize that benchmarks are by definition artificial to varying degrees.   There is, IMO a “continuum of value” in benchmarks – a reflection of how close that benchmark is to a real-world use case.  

In some cases where “single scenario” benchmarks (like SPEC SFS) really have significantly less relevance.   One example would be the very large scale, very mixed workloads we see at the large scale enterprise array customer use cases.  For those there is really a demand for a new type of storage benchmark.   These can be VERY relevant if that’s the only workload you put against the array – but more often than not, customer blend workloads.   At best in that case, the “single workload” benchmarks can hint at architectural goodness/badness (but doesn’t correlate 1:1).

More on this topic in a blog post tomorrow.  For example even with a “NFS workload” there are differing types – for example, VMware on NFS worklods.  SPEC SFS is a good indicator of more traditional NAS workloads.

The ideal benchmark IMO would be kind of “VMmark-like” in the sense that it would need to be a composite of many “real world” workloads, and would need to be able to scale to very large numbers of hosts, all very random.   Those kinds of workloads are very tough to simulate (look at how hard it is to do a really, really good VDI workload).

All that said – I’ll reiterate… you CAN expect lots more on the public benchmarking and benchmarketing from EMC – feedback on whether it’s good or not is welcome.

What’s the nutshell on this one?

• The EMC VNX performed at 497,623 SPECsfs2008_nfs.v3 operations per second – a new SPEC SFS record.
• That result is significantly faster than the previous leaders: 49% faster than an HP system (BL860c i2 4-node HA-NFS Cluster), 23% faster than an IBM system (IBM Scale Out Network Attached Storage 16-node cluster, Version 1.2), and 160% faster than the NetApp FAS 6240 Series system.
• The EMC VNX Series also had the lowest overall response time of systems tested, taking the top spot with a response time of .96 milliseconds. 
• The major factor in this result was the fact that we’ve been very focused on being able to leverage (not just use, but scale to squeeze the most out of) the very large multi-core performance of Intel’s Westmere tech, and couple that with very pervasive use of SSD (as both cache and a tier).   The amount of R&D focused in VNX-land on how we scale up by adding more blades that handle NAS or block duties, how each of those scales up as more cores are added per blade is as much of an engineering effort as the parallel efforts in VNXe-land around being able to run all those functions on one kernel and one blade.

If you want to see EMC’s result (and everyone else’s) click on the link below.

While “world record breaker” is the headline (and that’s still only using 4 active datamovers where we could go to 8, or 7 with a standby, and could have added a lot more SSD and westmere cores on the backend if we wanted to) – there’s a very important story behind the story.

If you’re interested in the insider engineering half of the equation read on….

So – does this really matter?

In one sense, no.  No customer runs SPEC SFS :-)  It’s also clearly a “lab queen” (a config built to show a huge result).   BTW, expect all sorts of competitive responses on that front.   Hint: EVERYONE’S SPEC SFS and SPC config is a “lab queen” in their own benchmark submissions – if you think any vendor doesn’t pull out the absolute stops, you are assuming they are lazy or stupid :-)  

However, in another important sense – does it really matter = YES.

The “yes” is that it highlights the incredible headroom and power in the platform.   It also highlights how much we can leverage several overwhelming trends:

• the emergence of massive multicore x86 systems (this config had more than 50 Westmere cores humming away)
• loaded to the gills with low-cost RAM
• attached via high bandwidth pervasive 10GbE and on the backend via loads of 6Gb SAS
• leveraging not a “touch of” SSD, but a “ton of SSD”.  This may seem “extreme”, but remember that what is “extreme” today is “mainstream” tomorrow.  

SSDs for example are on the steep part of the mass-commoditization curve.   I’ve purchased them (at EMC cost for various VMware/EMC things) for the last 3 years so have been a direct witness to what’s happen.   What a 200GB SSD (with crazy high IO perf) costs today vs. 3 years ago is ridiculous.   The amount of innovation in the consumer SSD space also has all sorts of volume effects throughout the industry.

So… Here’s a “peep behind the curtain” from a “industry insider”.

about 3 years ago we had a “oh my goodness” moment, where we realized two big things:

A) There was a wave of technologies that would reshape our “core” business of storage:

1. Yup, x86 was going to win. 
2. The impact of massive low-cost multicore was going to rock us – as our codebases as they stood weren’t able to scale across tens of cores sufficiently well. 
3. Mass 10GbE (and more importantly the staged nature of transition itself) would reshape host connectivity. 
4. More than anything else, the split of non-volatile storage into incredible performance (SSD) and massive capacity low-performance (magnetic media) would reshape everything we’re doing.
5. There would be an inevitable trend with ideas of “scale out” becoming important in some use cases.

These conclusions led to a lot.  1) led to the Symm being completely rearchitected.  2) led to a TON of changes in Enginuity, FLARE, DART.   In the FLARE/DART side, many of the innovations in FLARE 30/DART 6 in the earlier gen were there to “burn those in for the the VNX HW generation”.  3) led to Ultraflex modular IO – something the rest of the industry is starting to do also. 4) led to massive investment in what it would take to support an array that was dominated by hundreds of SSDs and the rest huge slow SATA/NL-SAS, both in terms of core codebase and system architecture (imagine the differences in the IO flow in terms of cache, latency and more), and features (use of SSD as cache and as a tier, and the clear necessity for auto-tiering); 5) led to organic innovations, acquisition plans, and looking at how to manage more with a single pane of glass (Unisphere)

But there was another realization.

B) we hadn’t invested enough in previous years re: cross-family synthesis/selection – and the degree of diversity was starting to hurt.  Further, the rate of both internal organic and external in-organic innovation was not slowing down.  Having different architectures is OK so long as the benefit outweighs the downside.   In certain cases, this is NOT the case, and the differences outweigh the benefits.

I talked about this during my VMworld PEX keynote (in the “censored out section” here).  I’m obviously not going to post EVERYTHING I talked about in the censored section (was for EMC/VMware partners only), but here’s how I phrased up the problem…

If you look at the customer demands on us, they have some commonality, but aren’t the same across the board.   For example, simplicity is a universal customer requirement.   But, whereas in the smaller customer segment, it’s right at the top, in the larger customer segment, while simplicity is important, if it doesn’t meet the performance/availability requirements – forgeddaboutit.

And those customer demands translate into diverging R&D priorities.

But the question in the red was the thing we really grappled with.  

If we didn’t make big bets on that front, we’d find ourselves years later with great performance, functionality, availability (all things that EMC’s brand is associated well with), but without cross portfolio consistency, simplicity and low COGS (cost of goods sold – which affects price and margin, very important for both our customers and EMC’s shareholders).

So – after about a year of debate (and debates never stop), we embarked on what I’ve heard Pat Gelsinger call our “4 year journey”.   We did it all relatively quietly, but we’re now well into year 2 of 4.   The net is that we have engineering teams in our Unified Storage Division working in a “producer/consumer” model where each are both producing innovations and consuming the innovations of others.  

This huge VNX SPEC SFS result is a manifestation of RIGHT side of that set of innovations.   Getting to that were several years of investment around those customer priorities.  Huge optimizations in the DART and FLARE stacks to be able to get the most out of the commodity hardware revolution in the dimensions of performance.   We were able to do that while accelerating the pace of innovation of functionality, and without risking/breaking much of the mature code stack’s hardened and proven availability.

The VNXe is the manifestation of the LEFT side of that set of innovations.  Getting to that were several years of investment around the C4 codebase, Unisphere and more, which “consume” the FLARE and DART IP from the other team but do it without two kernels, and separate hardware.   Most of the innovations around performance, functionality, availability you see today and tomorrow in VNX shows up in VNXe with a little time lag.

But – it’s a bidirectional street.  The things you see in VNXe show up in VNX (and to a lesser extent across the EMC portfolio) with a little time lag.   The C4 codebase is extensively used already in VNX, and accelerates innovations in the core FLARE/DART stacks.

It’s a really neat thing to be witnessing, and you’ll see a LOT more announce and ship around both in 2011.

It’s REALLY hard work, and my hats off to the engineering org.   We are far, far from perfect.   We are, however, on year 2 of the journey – and the early sacrifices are starting to have their payoffs.  I can feel the acceleration happening all around me. 

As always – courteous (including critical) comments desired and welcome!

Comments

[paul]

hi VG, just wondering is there any utility you would recommended for monitoring san performance ? i've tried storage profiler but i was wondering if there are any other packages out there

regards
paul