What's in a study?

EMC's been proudly showing data from the last two year's "IDC Server Virtualization Reports".   Recently, an IDC person was present when I showed the pie chart, and he corrected me: "it's not market share" he said.   I felt awful.   I asked him how he would call it.  He said he didn't know - but it wasn't a market share study.

So - let's see if I can be an engineer about something that isn't engineering, but is marketing.   Read on...!

So - every year, IDC does a study on Server Virtualization that gets released in the latter part of the year.   I'm not an expert on surveys, but it seems pretty fair.  No single vendor can buy their way in, although often several vendors sponsor it in part - but none so much that it has an influence on the outcome.   The methodology is totally removed from the vendors.   For example, EMC partially sponsored it in 2006, but not in 2007 (VMware partially sponsored it in 2007).

Below are the covers of each report.

 

 

From this, EMC (lump me in here) extracted this pie chart (I've circled the incorrect part) which is the source of the controversy, and was presented at EMC World 2008:

Now, each one is a survey, and all surveys have critical methodologies. 

Here's the Methodology from 2006:

• Sample of 405 Senior IT Managers/IT Directors that have implemented server virtualization within their organization
• Are involved in recommending, purchasing or approving servers and/or server operating systems
• Are involved in implementing virtual servers or familiar with the details of their virtual servers
• US Sites
• 100+ employees worldwide
• Sample is almost evenly split among those who responded for x86 and non-x86 virtualized servers
• x86 Architecture N=241
• RISC/CISC Architecture N=164
• Sample was drawn from NetworkWorld subscriber list
• Interviews were conducted during July and September, 2006
• Margin of Error = +/- 4.9%

Here's the Methodology from 2007:

• Sample of 410 Senior IT Managers/IT Directors that have implemented server virtualization within their organization
• Are involved in recommending, purchasing or approving servers and/or server operating systems
• Are involved in implementing virtual servers or familiar with the details of their virtual servers
• US Sites
• 100+ employees worldwide
• Sample is almost evenly split among those who responded for x86 and non-x86 virtualized servers
• x86 Architecture N=214
• RISC/CISC Architecture N=196
• Sample was drawn from NetworkWorld subscriber list
• 35 minute phone interviews conducted during October/November, 2007
• Margin of Error = +/- 4.9%
• Additional 78 surveys were completed with “Non-Qualified” respondents

Ok so immediately a few things jump out at you - it's US only.   You can't draw strong conclusions from this outside the US.  Second, I'm not sure where a survey crosses into a representation of market share - but it's a pretty solid sample size - 400+ respondents.   Also, not all of the respondents were running VMware - rather some form of server virtualization.   Using the data from the survey though (there's a TON of great info in there) - you can extract a pretty good idea of what percentage are using VI2.5, VI3.x, or other virtualization products.  I can tell you that x86 server virtualization (which was 200+ of the samples in the survey) was dominated by VMware.

So - how did EMC produce the pie chart.  From this data...

Sept 2006 (read the caption!)...

Dec 2007 (read the caption!)....

That caption is great, I love it :-)   It was so inflamatory though, that we didn't issue a press release (we seriously had one drafted up) for fear of pissing off too many of VMware's important partners.   Note that it's not saying that IBM/Dell/HP's server attach to VMware is changing, it's just that the choice of storage is becoming decoupled from the choice of the server.   The survey (of course) has the statistics on the server attach too - among a lot of other cool stats.

Now - there's two things that if your EMC salesperson (they are wont to hyperbolic chest-beating) tells you - you should correct him fast:

• "EMC dominates worldwide storage attach for VMware"  - I don't know of any statistics beyond these (Gartner, or others - if other surveys have been done, I would love to see them) - but one thing you can be definititive on - the data is for the US only, since the survey was US only.
• "EMC has dominant market share for VMware"  - this is where we erred, and IDC corrected us.   In that presentation, I when I asked the IDC fellow what he would call a survey of 410 customers - and he said "a survey" - so that's what we're calling it going forward in all our content "IDC Survey Results", and have asked our sales folks to update their presentations.

But likewise - if another vendor tries to tell you "EMC is incorrectly using the IDC report for false purposes, and WE are the dominant storage vendor" - this is the honest insider's perspective:

• There is no conspiracy - but rather an simple honest mistake from people at EMc who are not analysts.  
• The data is correct (plus or minus 4.9%).  
• The sample is 410 customers in the US.
• If another vendor has a study contradicting this, man, I would love to see it
• Utlimately, the error is the title - it's not "Market Share", but rather "IDC 2007 Study Results".

Ok - after that short interlude - back to good interesting technical stuff :-)  Comments welcome - from customers, analysts, competitors.  Is the representation I'm proposing fair?

Comments

[Jason]

Is this IDC research available for public purchase? I've been searching their site and can't find it. Do you know the report number?

[Chad Sakac]

Jason - thanks for the question, am going right to IDC to find out about purchasing the report, or if I can post the softcopy. If you don't hear from me in a day or two ping me again (I get an insane volume of email, and sometimes I drop a ball - I'm interrupt-driven).

[Virtual_JTW]

I too would like a copy of the survey if it's available...