Virtual Geek

"Physical servers kill baby penguins... go virtual"

ROTFL!  

Clearly not the work of a marketing genius, but this one had me smiling.   I can't take credit - was EMC's Derrick Baxter - the guy who heads up a huge chunk of our Exchange on VMware testing here at EMC.   He was part of the original Exchange Ranger (now called Microsoft Certified Master program) rotation when he was at Microsoft, so to call him "a vet" is being woefully understated.  They also trained those folks to be hyper-conservative - ultra "do it this way and it works".     I'll never forget when VMware and EMC started the first test - he was.... let's say... skeptical.

Now the title of this post is his hotmail sig.   The power of virtualization with VMware has made another convert :-)

Oh - BTW - if you missed it, we just added to the Joint EMC/VMware Reference Architecture library (this one done by the OTHER crack Exchange squad at EMC in Cork Ireland)...

"Metropolitan Exchange Recovery in a VMware Environment" - H5805, available on Powerlink.

We have solutions where we have protected via CCR (because sometimes that's the right way), via SCC - like here (because sometimes that's the right way), and via SRM (because often that's the right way).

Oh, while I'm at it, I've said it before, I'll say it again... you've got to love laughing... and today I got two doozies...   The counterpoint to VMware's vExpert program, Stu's vIdiot program :-)

VMware View Composer/Linked Clones - they are NOT a panacea

May seem strange to some of you for me to post this RIGHT after I do a post on the VMware View Reference architecture :-)

It's not, however.   There is no IT panacea.   There is only technology to be applied to problems.   And, generally, I think it's about know-how, not technology itself.   Don't trust people who say otherwise :-)

Ok - what are we talking about?   Ok, got a question from a very advanced VMware shop today looking at this and examining the blogosphere for info.    OK, background reading...

First - a good post on this here from Stu at vinternals.

Second - a good rejoinder/ongoing commentary here from Rodney.

Take a read - then come on back and read on...

VMware View Manager/Composer 1000 Client Reference Architecture

A shout out to my compatriots on the VMware View team, and congrats on the VMware View launch two weeks ago... Congrats!   I've been heads down with our Quarterly Business Review and Quarterly Technical Reviews at VMware this past week, so a bit delayed on this post.....

We've been slaving away on a lower-scale VMware View building block, something more broadly useful for the last couple of months at the Santa Clara Lab I mentioned here: ; and where we have done 10K client scaling testing and 250 user building block VDM 2.1 reference architectures I mentioned here.  

BTW - If YOU are looking at a large scale (let's say 5K clients or more) Client Virtualization project, reach out to your EMC and VMware account teams and ask for some time with the engineering teams in Santa Clara - we built it to openly share our learning.....

Back to the subject at hand... So, I'm happy to say you can get it now - a VMware View Manager/VMware View Composer 1000 client Building Block Reference Architecture.   Read on for more!....

UPDATED - 9/13/2009 - Some interesting VDI performance findings and updated Use-Case Solution Guides

While much of the discussion and attention will be focused on the very high end VDI scaling cases next week, I thought it would be useful to show some stuff at the other extreme.   We also have been testing in the 125-250-500-1000 user ranges on our NS20, which is an mid-range unified system.   This is something someone could get started in the $30K range.    Note - I'm not talking about what we're doing at the other end of the extreme - the 10K+ use case (we'll talk about that next week).

There is a detailed document that covers these best practices, H5719: "EMC Infrastructure for Deploying VDI in the Enterprise".   The document is on powerlink, and will be up on EMC.com shortly.  The full validation test report showing all the performance envelopes will be up right after VMworld on powerlink.

UPDATE: Folks, the NetApp side has made some comments that in their view we're not being clear here: http://blogs.netapp.com/virtualstorageguy/    Vaughn is a good guy, so I'm going to take the personal slag (ouch - used car salesman) in stride.   We are replicating the LUN objects in this case, and they are the unit of replication.   If you will have more than roughly 200 VDI VMs per cluster, you need to have several VMs per LUN.   The PowerVDI tool automates this.   These ratios mean it's more accuate to say "you will invariably achieve 100:1 space savings, and may achieve more depending on the scaling goal". (the ratio of source VMs per source LUN needs to be higher at higher scales, and therefore 100:1 is the worst case model).   There is NOTHING intrinsically wrong with NetApp's approach, there is NOTHING intrinsically wrong with EMC's approach.   Each have advantages and disadvantages, and are a function of each vendor trying to solve the customer problem.  The ratio of LUNs to VMs is not 1:1.   We don't do this because we love block - while we can take 96 filesystem snapshots, only 16 of those are writeable, so this approach is better for EMC customers. Just setting the technical record straight.

If you want to see some interesting performance metrics - read on!

Virtualizing Sharepoint - How about saving 74% of your power?

Folks - we have talked a lot in the past about Exchange 2007 workloads on VMware Infrastructure here (net: Exchange 2007 on VMware works GREAT, you can get more users per server than you can on physical, you can get all the goodness of the new Exchange 2007 features without the server sprawl they create, and still get all the storage savings).   Some folks have asked where they can get the detailed doc, including performance data.  Read on for directions on how to find ALL the solutions docs(i.e. "teach someone to fish") - but if you're lazy, just download it here (also to just to stop the "EMC doesn't share detailed info" doubters out there). 

BTW - there is more to come on that front - we still shattering our own current records of 16K users per ESX server (configured as 8 mailbox server role VMs each with 2K users) as new server platforms hit the market (last round was with the R900 series server with 64GB of RAM).

Now on to another "Tier 1 app".... Sharepoint is another great app, and a great app to virtualize since it's one that often has a lot of server components (like like Exchange 2007) and people were asking about our experiences running it on VMware.   Curious?   Read on....

Array Snapshots in the VDI Use Case and performance scaling

Phew - after all the politics and answering all the speculation with my own speculation, it's nice to get back to my comfort zone - a good technical topic. 

This is a question that comes up pretty often.   "OK - I want to save money on storage, accelerate image deployment and improve image management".   EMC and a few (and it really is a few) other vendors are looking at integrated VDI solutions leveraging VMware's capabilities and the array ability to take pointer-based writeable snapshots to save $$ on the backend (and that's "so far" - wait till you can see what we can do in the future :-)

You can see a video of what I'm talking about here:

and a whitepaper describing it here: http://www.vmware.com/files/pdf/partners/emc/emc_virtual_desktop_for_vmware_solutions_overview_wp.pdf

The inevitable question is - what about performance scaling?   All those VDI clients are hitting an original set of source blocks on the underlying storage subsystem.   So - what do we find?   Read on to find out.....

Answers to a bunch of questions

I got a lot of questions on the 10GbE post, which is great.   Some of the answers involve graphics, so rather than comments, I'm just going to do a post....

Here are the original questions:  http://virtualgeek.typepad.com/virtual_geek/2008/06/10-gigabit-ethe.html#comments

Ok - one at a time (apologize for the delays, have been swamped):

To Duncan: 

jumbo frames don't hurt, but don't help that much either.   Below is a table from some pure research with an Exchange 2007 ESRP workload (handy because it exhibits both small block 8KB random I/O during the normal workload, and large block (64K and larger) sequential during the checksum and backup phase of the test.   I also like it because it's a app workload, not artificial.  This test was with VMware 3.0 with the MS iSCSI initiator in the guest, but similar more recent tests show the same behavior using the VMware iSCSI SW initiator in 3.5.

J1 = Jumbo frames on; J0= off.

Takeways: for the IOPs-driven small block workload - the difference is a fat nada.  For the throughput large block workload, there's some difference - in a 4hr backup job, it shaved off 12 minutes.

I'm going to do an IP best practices post (get a lot of question re: multipathing configs) and I'll post the data we found.   Jumbo frames **CAN** hurt you though in the sense that you need to have the discipline to make sure you enable them from the source to the target, otherwise you get fragmented and those nice iSCSI PDUs get sliced up incorrectly.   What's more important is something basic - design the IP network like you would an FC SAN.   Isolate it (phyiscally or via VLANs), and build in redundancy.   Don't route it.  Also, disable the Spanning Tree Protocol on the ports used for iSCSI (or the switch as a whole).   This Cisco doc says it best:

http://www.cisco.com/en/US/prod/collateral/ps4159/ps6409/ps4358/prod_white_paper0900aecd801352e3_ns512_Networking_Solutions_White_Paper.html

"For the purposes of creating the iSCSI test environment, it is recommended to keep the configuration simple and assign an IP subnet that is to be used for both the iSCSI initiator (s) and the Cisco MDS 9000 Series IP services module's Gigabit Ethernet port"...

Ah - KISS.   Now, where I have seen iSCSI problems is in high-thoughtput (usually streaming backup) workloads.  For this as I said earlier, multipathing (VMware ESX, or for that matter any OS) is important.  You should have multiple iSCSI targets configured (VMware doesn't support multiple connections per session yet with their software iSCSI target, and depends on multiple target IPs for network level load-balancing, or multiple targets for storage-level multipathing).   EqualLogic does this by default (in their case every LUN is an iSCSI target), NetApp, Celerra, CLARiiON (they have LUNs behind iSCSI targets) all require a bit of thinking by the admin (but not a lot).  The Celerra is particularly sweet - you can configure up to 1000 iSCSI targets.

To Michael:

I hear you loud and clear, Michael, and IB is an engineer's dream of a transport.   To tell you the truth, I don't KNOW why it's not compelling, but it seems to have been relegated to the sidelines (certainly not by what I post), but rather by the market as a whole.   For example, there's a startup called Xsigo that has a really cool IB switch and have made a IB CNA focused specifically at this VMware use case.   Yet, in spite of all that goodness, I haven't seen it go far, and have been following closely.  My theory is that it has more to do with the cable plant - which is why I suppose (and this is 100% supposition) that 10GbE won't take off (i.e. capture the mass market and rapid adoption curve) either so long as the physical layer is an optical cable.   That train of thought is what made me ask my last question: does it have to be CAT6, or would something that is similar (like the SPE+Twinax) be good enough, or - is this a solution in search of a problem (I don't think so, which is the basis of the post).  Why do you think IB hasn't taken off?

To Charlie:

Charlie - you know me, I will never lose my iSCSI passion :-)   This new FCoE hasn't even tempered my thinking on it - I'm still (and will always be) mr iSCSI.   Like I said in the post - **iSCSI WILL BE THE MARKET MAJORITY OF BLOCK IN THE FUTURE***.   Heck, I even have a standing bet with Chuck and others on the exact degree of market share dominance by 2012 (and I say that by any measure: capacity/ports/revenue).   What I'm saying is that for some smaller (but very important) portion of the market, the need for lossless transport is a deal-breaker, so something will need to coexist with iSCSI.   I see your argument - that convergence is the full stack, but I haven't heard that from customers - I've heard that to most, it means the CapEx.   But, let's be clear - Ethernet as the physical/link layer is a given, and iSCSI immediately makes that a converged solution, period.

To Ole:

Ole, of course, we've tested it a million ways.  Long and short:    iSCSI works well with IOPs focused workloads with normal latency requirements.   in the 8KB IO size wihout jumbo frames, it adds a few percentage points of overhead (i.e. iSCSI PDU fragmentation with the standard Ethernet frame size and associated TCP/IP overhead).   A few percentage points is not a big deal.   It's higher on large I/O sizes (64KB and larger), but still small enough that most people don't care except in the academic sense.   

But you're right - the point isn't that iSCSI is routable - routing iSCSI is a BAD idea.  Storage expects latency that's measured in milliseconds if block, and hundreds of milliseocnds if it's NAS (of course NAS can have millisecond latencies also, but as a protocol, it's designed to expect and operate with more).  Ethernet switches add latency characterized by microseconds.   This means they are effectively invisible from a performance standpoint.    Routers of course add latency in milliseconds.  iSCSI is a block protocol, so you do the math, and come to the conclusion you did - the routable thing isnt' why iSCSI is successful.

We've done versions of the table I posted earlier - here is a recent (joint EMC/VMware solutions testing effort) big brother at 8 times larger scale:

 

Here was the ESX view of the storage subsystem - note the IO reaches around 16K IOps, but maxes out at 100MBps - iSCSI piece of cake:

 

Here was the CLARiiON's world-view - the SPs were pretty busy, and I can tell you that with this workload, read cache is totally useless - the host-generated IO is almost exactly the backend I/O:

 

So, iSCSI (if you design the iSCSI network like an FC network), performs fine, the encapsulation and Ethernet frames add a few percentage points of overhead, and adds one or two milliseconds - not bad.   The processing power?   We've found that to saturate a two full NICs with iSCSI burns about 1 core on a modern multicore CPU.   For those of us in the era of processor-bound systems, you look at that initially as a big deal, then think about it and go "so what - buying a quad core vs. a dual core costs me $100"

Now - what about a throughput dominated workload like Business Intelligence (or an aggregate guest-based backup from an ESX server with 20+ VMs on there?  hence Avamar).   In that case, throughput matters, and while eminently possible on iSCSI, with 1GbE, it's a bit unwieldy - I mean, who is REALLY going to have a bundle of 8-10 cables just for IP storage per ESX box?   Think about the cabling mess as you scale that up.   Now... 10GbE - another story entirely.

BUT - you are right in the sense that you if you have an existing FC infrastructure (both hardware and people versant in WWNs), iSCSI solves a non-existant problem for you - which is why in spite of it's incredible growth (36% CAGR or more), there is very little iSCSI in larger enterprises.  Not because it's not good, but rather it solves a problem they solved before.

It brings me back to my original view - if you have the FC infrastructure (hardware and knowlege), don't underestimate the power of being able to ping your storage target.   If you do, what will be the impetus for the next change (change being inevitable).   I think it's that VMware and mass consolidation, in it's second wave is going to make 10GbE mandatory just for networking, which then leads to "ah, just converge the IP and Storage networks dammit!"

to Brian:

Brian - thanks for the input - I'm curious what others say.... I'm surprised that there hasn't been more of a move here already towards and optical plant.  But, most enterprises are still very much using Cat 5e/Cat 6 even for core uplinks and aggregating/trunking like made.  Heck, Verizon runs FIOS to the home :-)    It just makes me wonder if there is a cost threshold that must be crossed, or perhaps as I think, it's more basic.   People who are networking people are comfortable with twisted pair - and it's less logical than it is emotional (we are all human beings after all :-) 

Oh, and what does a Corporate VMware Infrastructure Lab look like?

After looking at my home lab (Building a home VMware Infrastrucuture Lab). This is my OTHER VMware lab during my day job:

This is my joint EMC/VMware lab in Research Triangle park.   It and it's major sister sites (Hopkinton, MA; Santa Clara, CA; Cork, Ireland) are part of our distributed Solutions Office operation - where we test not interop (that's a much, much bigger lab called eLab - where we do core interop testing and things like run the our gear through VMware HCL cert harness).

So what's in this lab?

• more than 450 ESX servers, each running the latest builds (and of course, beta builds).  We keep around some physical servers, but really just to re- baseline against new ESX server builds (i.e. rebaseline physical to virtual and previous virtual rev to current virtual rev).
• more than 40 Virtual Center hosts, Lab Manager, Life Cycle Manager and Stage Manager.
• more than 40 Arrays (both EMC and non-EMC) - more than a PB of storage
• It's not done at this site, but in Santa Clara (picked for closest proximity to Palo Alto) is where we do our VDI testing - and have a rig designed to scale to multiple tens of thousands of clients, and where we have VDMv2 and all our integration there.
• We have totally automated harnesses for Exchange 2007, SQL Server 2005, Sharepoint Server 2007, Oracle 10g, 11g, SAP (and many many others)
• We use a wide variety of load-generation mechanisms, from LoadGen to Quest Benchmark factory to ORION
• From a human standpoint, there is a team of around 60 people at EMC that work on these labs non-stop (literally 24x7 - because we split work shifts around the clock).

It's hard to tell in the photo - but the floor is BOWED - this is on the 2nd floor of the building in RTP.

I can say that last year, the solutions validation testing around VMware was rough and tough a $30M effort.

The purpose of these sites is to REALLY test.   Build docs that aren't marchitectures, but include failure characterization (i.e. total envelope testing).   Test all the functional use case scenarioes (i.e. how customer use our products).    

The key is that we do this at a wide variety of scales: hundreds of users to tens of thousands, client counts in all ranges, and all sorts of database workloads.   Why - every customer is different - we have solutions targeted at customers getting started (heck, even consumers now), all the way to the biggest of the big.

We don't publish all the docs publicly - which is a policy I personally disagree with.   Why?   Because competitors are geared to compete with EMC as the defacto "800 lb Gorilla", and rip apart the failure cases (news flash for anyone who does this for a living - everything has a failure limit - the name of the game is to know WHEN a given config will fail, so we can customer minimize customer pain) and build competitive material on it.   Sounds, reasonable?   Not to me.   I think we might as well be wide open with it - I'm not afraid of describing why a scenario failed in a given workload or a given config.   I think we should publish it all.   What do you think?

Here are some of the lighter (and some of these ain't so light!) examples of doc packages (we usually publish a "reference architecture", an "applied technology guide", but don't publish - except internally for customers, partners, and EMC on Powerlink - the big-daddy - the "validation test report"):  http://www.vmware.com/partners/alliances/technology/emc-whitepapers.html

There are also extensive Partner Engineering center - i.e. there's one in Redmond across from the Microsoft Campus, and one in Santa Clara for VMware.   Santa Clara isn't just where we do VDI, but anything that's very intimate with VMware.   That joint work resulted in this:  100,000 I/O Operations Per Second, One ESX Host

Long and short - MAN, we take VMware very very seriously - and are making the investments in both people (important) and gear (less important - but still important)