Virtual Geek

A “Multivendor Post” to help our mutual iSCSI customers using VMware

UPDATE: Sept 21st, 2009 – 4:24pm EST – Are you using vSphere, or thinking about it?   We’ve done an updated “Multivendor iSCSI post” with vSphere details here.  If using ESX 3.x read on…

Today’s post is one you don’t often find in the blogosphere, see today’s post is a collaborative effort initiated by me, Chad Sakac (EMC), which includes contributions from Andy Banta (VMware), Vaughn Stewart (NetApp), Eric Schott (Dell/EqualLogic), and Adam Carter (HP/Lefthand), David Black (EMC) and various other folks at each of the companies.

Together, our companies make up the large majority of the iSCSI market, all make great iSCSI targets, and we (as individuals and companies) all want our customers to have iSCSI success.

I have to say, I see this one often - customer struggling to get high throughput out of iSCSI targets on ESX.   Sometimes they are OK with that, but often I hear this comment: "…My internal SAS controller can drive 4-5x the throughput of an iSCSI LUN…"   

Can you get high throughput with iSCSI with GbE on ESX?   The answer is YES.  But there are some complications, and some configuration steps that are not immediately apparent. You need to understanding some iSCSI fundamentals, some Link Aggregation fundamentals, and know some ESX internals – none of which are immediately obvious…

If you’re interested (and who wouldn’t be interested with a great topic and a bizzaro-world “multi-vendor collaboration”... I can feel the space-time continuum collapsing around me :-), read on...

"Site Recovery Manager in a Can" - Celerra Virtual Appliance HOWTO 401

UPDATED Jan 19th, 2009 - the updated doc (v1.01) reflects SRM update 1, general minor corrections, and the removal of the ONLY step (Solutions Enabler) that required an EMC powerlink login.

Folks, here it is - the "all-in" step-by-step guide for setting up Site Recovery Manager with nothing more than two ESX servers (which you can even do with home-brew servers, as noted here) and the Celerra VSA.

If this is your first read, you can either just click on the document below or start the 101, 201, 301, 401 journey here.

This guide isn't the first one of these out there.  I think my respected friend Adam Carter at Lefthand did it first (Adam, hope you're enjoying HP!).  The NetApp one is pretty good too.   What I like about ours is the extreme completeness.   It's also nice that the Celerra VSA has no timeouts, no limits (except that you'e not allowed to use it in production).

A quick shout out...   This was absolutely yeoman's work by one of the newest members of the Global EMC VMware Specialist squad - Bernie Baker, with strong assists from one of the other newbies - Stephen Spellicy.   Both of these guys started relatively recently, and this work is (IMHO) FANTASTIC - it highlights that this stuff can be EASY (and that they have what it takes to have a early impact).  Bernie - you've set the bar high for the team and yourself - GREAT WORK!

Until the next major revision of the Celerra VSA (which will be linked to major feature releases), expect to see only one more "301"-level Celerra VSA post (downloading and using EMC Replication Manager for VMware for point-in-time, VM-consistent instant VM backup/restore) - we're warming up in the batter's box a series on Avamar Virtual Edition....

HOWTO - Benchmark an iSCSI target (and any block target, including within VMs living in NFS datastores)

Someone asked me out of band  (thanks for asking, MattL) what I would do to quickly get a envelope of an iSCSI target.  

In all the post-VMworld followup, I don't want to do a long dissertation here, but quick off-the-cuff comments may be useful.   FYI, these are the quick set of tests we do during the application validation testing to quickly see if we're on the mark before we start with app workloads.   Read on for the details!

Welcome - My friend, NetApp's Vaughn Stewart

We may face each other often on the battlefield, but I relish every battle :-)   My good friend an colleague Vaughn Stewart has joint the blogosphere - check it out, and add it to your reader!.   Vaughan - welcome, and drinks are on me at VMworld!

Vaughan, I love ya, man - but come on (seriously) on TR-3697  - it's like something Chuck would make :-)  I love NFS storage for VMware - it doesn't get the love it should, I love iSCSI storage for VMware, I love FC storage for VMware, I love FCoE storage for VMware (and on this one, although I literally have at least a few grand riding on iSCSI, I've come around to the "it's not a conspiracy" view like Dave has here - though "wait" is subjective - VMworld is right around the corner).   So long as it's storing VMs and the customer is happy, I'm happy. 

But seriously....

"The goal of this testing was to compare the throughput capabilities of ESX 3.5 with NetApp storage using FC, iSCSI, and NFS protocols in a real world environment."

Ok, if that's the case, what's the scoop with:

• 4K/8K IO size only
• 2Gbps FC
• You guys have "throughput/IOPs" shown only in relative, not in absolute.
• 84 144GB drives with 16 VMs driving the IOMeter workloads with * 10GB of data each on them =  1.3% utilization (rounding up!).  

This is a workload and a comparison chart that Netapp dreams of - but unfortunately doesn't match the noble testing goal.   I love NetApp filers.   I own them.  I have bought them with my own $$.  What they do they do extremely well - but WHY do you guys insist on running workloads in these benchmark docs with these crazy low utilizations??

WAFL = greatest strength in some use cases; greatest weakness in others. That's not BAD, that's engineering.   Engineering is all about tradeoffs.  

WAFL's tradeoff is free snapshots in exchange for non-linear performance under normal workloads and normal utilization.  It's your superpower, and your kyptonite.   Kickass write perfomance when there are loads of free spaces to write to.   Another manifestation is post-process deduplication - but not not pre-process (which no one has figured out for production storage but is our goal) which is really another GREAT manifestation of WAFLs underlying characteristics)    Good read performance overall.  Poopy write performance when it's random with a full and fragmented filesystem.    

BTW - do NetApp filers catch fire under this condition - NO, but they do slow down.   In the same way our arrays don't catch fire with snapshots (as much as you guys say they do).   But both are designed for different tradeoffs.

Again - that's not bad, that ENGINEERING - it's all about trade-offs.    EMC has many of the same sort of things, in different places.   You guys make great platforms, so do we.  

Come on, let's not be Chuck here.  If you wanted to be real-world - let's do a broad set of IO sizes... you're a smart dude (you REALLY are) - so let's see it at 64K (the current max, larger than that and ESX starts chopping them up.   Let's see that a 10%, 20%, 50%, and 80% utilized.   Let's see it after a few days of random workload.

I would expect you would see the same thing I see, and the same thing VMware sees (this is covered here - btw, this doc is flawed also - broad IO ranges, but 100% sequential write and read only - but I don't feel bad, those are workloads that favor WAFL over traditional block layout schemes):   

Let me summarize (both TR-3697, EMC's docs, and VMware's docs agree!):

NFS and iSCSI are fantastic and near FC in throughput and IOPs on small (4/8K) IO workloads (though consistently higher in CPU utilization - though MHz are close to free these days), but they diverge (this is not bad, it's expected) at large IO sizes, and large througput workloads - even with 2GBps FC (which is the chart above from VMware) - which no one even buys anymore.  This is different with 10GbE and jumbo frames of course.

When you publish a benchmark that shows "we filled this to 70%, ran it with a random write workload for 3 days, and THEN ran the benchmark, and we compared it with current FC technology", you can come over for dinner :-)  Ok, screw that, I'll buy dinner AND drinks at VMworld!

Ok, that off my chest, I will NEVER let my blog (or me) devolve to being anti-NetApp.  You guys are a great company, a great competitor, and are filled with great people.   I relish competing, because it brings out the best in both of us (except silly stuff we each do to compete with each other) and that's good for us both, but more importantly, good for customers.  I've said it publicly before, I'll say it again:

My comment:

"For what it’s worth – personally, certainly I don’t necessarily represent EMC in this context – I appreciate having as formidable a competitor as Network Appliance. With other competitors, we compete with them, and it’s a slow waltz, but with you folks, it’s a tango. It makes us better, win or lose with any individual customer, and in the end, it’s good for the industry and the customers."

And Dave Hitz' comment:

"I completely agree. I feel the same way about EMC."

Celerra Virtual Appliance HOWTO 101 - Basic Setup and Configuration

Glad to see that people are having downloading and success with the Celerra VM on VMware Workstation based on this original post.  I wanted to provide a quick "HOWTO" to help, and will publish a 201 (setting up Replication and a 301 - configuring SRM).  

Most folks are doing well if they just RTFM, but I thought a little "hand-crafted walkthrough" might help others.   The standard documentation is geared towards VMware Workstation, which is the officially supported target (BTW, you don't need to ask, we're making ESX OVF a standard package :-), so some people stumbled a bit when trying to use it with ESX.

I've updated the OVF (some of the problems people were having were a result of the actual previous OVF I posted - SORRY - sheepishly) .  Make sure you download the new one if you're going to follow along (again, you can get that here)

Ok - have you got it?   Ok... then read on...

Quickly then here - in this HOWTO:

1. Importing the Virtual Appliance
2. Get the Celerra Sim up on ESX, and configuring the VMX
3. Configuring network interfaces (and what they all mean)
4. Making the Celerra Sim unique after cloning
5. Licensing Celerra features
6. Configuring Datamover interfaces and iSCSI targets
7. Getting iSCSI LUNs to the ESX cluster
8. getting NFS exports to the ESX cluster

I will post followups shortly:

• 201 series (adding storage to the simulator, configuring snapshots, configuring remote replication, etc)
• 301 series (configuring Site Recovery Manager, VDI mass replicas, etc).

In a little bit, I'm going to start posting bit by bit other EMC VMs (they are legion - ADM, Control Center, Avamar, Replication Manager, Networker 7.4.x, IT Compliance Analyzer, EMC Backup Advisor, and that's just getting started) with similar guides, so let me know if this is useful - BTW - they are also going through the formal release and VM Appliance programs, but I'm notoriously impatient, and think many of you out there are smart enough to get some fun out of these without too much hand-holding :-)

Read on......

Array Snapshots in the VDI Use Case and performance scaling

Phew - after all the politics and answering all the speculation with my own speculation, it's nice to get back to my comfort zone - a good technical topic. 

This is a question that comes up pretty often.   "OK - I want to save money on storage, accelerate image deployment and improve image management".   EMC and a few (and it really is a few) other vendors are looking at integrated VDI solutions leveraging VMware's capabilities and the array ability to take pointer-based writeable snapshots to save $$ on the backend (and that's "so far" - wait till you can see what we can do in the future :-)

You can see a video of what I'm talking about here:

and a whitepaper describing it here: http://www.vmware.com/files/pdf/partners/emc/emc_virtual_desktop_for_vmware_solutions_overview_wp.pdf

The inevitable question is - what about performance scaling?   All those VDI clients are hitting an original set of source blocks on the underlying storage subsystem.   So - what do we find?   Read on to find out.....

UPDATED (old VSA is retired!!) Get Yer Celerra Virtual Machine here!

UPDATED (2:30pm 4/1/2009) – there is a new Celerra VSA here.   I’m keeping this old post for the threads.

10 Gigabit Ethernet and VMware - A Match Made in Heaven

By now, it should be obvious to everyone (but surprisingly it's not) that VMware is driving 4 large scale IT trends from an Infrastructure standpoint.

1. A raison-d'etre for multi-core CPUs -  if you look, it's not uncommon to see 20:1 consolidation ratio with today's dual core and quad-core processors in the dual socket platforms (blade or rack-mount - that's for another post :-) ).   Heck with some workloads (VDI as an example), and 8 cores and lots of RAM and ESX-server's memory dedupe, much higher numbers are possible even today - 100:1 even.   If you look at the very near future - Nehalem will intro with 4-core dies - but will scale to 8-cores.  Each core will have two parallel execution paths (Anandtech does a great job of covering this as always: http://www.anandtech.com/cpuchipsets/intel/showdoc.aspx?i=3326&p=4
2. Consolidated IO workloads -   even before point 1 - even when you move from 1:1 to 20:1 consolidation of I/O becomes a core design bottleneck.   It's always shocking to me that people don't realize this.   Let's say you're an Dell shop, and standardize on PowerEdge 2970s.   How many NICs do you have in a standard physical server?   It's usually 2-4 (let's say it's 3 for the sake of being even handed).   When you consolidate that only a honkin' R800 with tons of RAM with ESXi, how many NICs do you have?   perhaps 12 (4 LOM, 2 x quad-ported PCIe NICs on the riser card).   Two go for service console redundancy (you ARE using a redundant service console, right?!?!), then you lose two (maybe one) for VMotion - so you have 8 left.   If you use either vmkernel IP-based storage (iSCSI or NFS), you will use some for IP storage, some for pNICs sitting off vswitches supporting Virtual Machine vNICs.   So quick math - 60 old physical NICs are now consolidated on 10 NICs (with the potential of less if you're using some for IP storage).
3. Shared storage becomes critical - This is true in a good way and a bad way.   It's true in a good way that you're bringing stuff that was on lower-availability infrastructure onto things that are more solid and consolidated, and that you're gaining the massive business flexibility benefits you get from VMotion/DRS,  Storage VMotion , VM HA, Site Recovery Manager etc.   It's true in a bad way in the sense that it's making it a cost prerequisite for many things that use to be a "C:\ on physicaldisk0" is now happily living as a VMDK on a VMFS or NFS datastore.   This is forcing all the storage vendors (EMC included!) to think about our infrastructure in a new way.
4. Management tools need adapt -  It just takes one thing to make this sink in.   Let's say you setup an SRM recovery plan.  Life is good.   Then you svmotion one of the VMs out of the container being replicated  - how valid is that Recovery Plan now?    Management tools need to integrate with Virtual Center, and have as a core premise that the app container is a virtual entity.   This has impacts that aren't immediately obvious - but perhaps more important than all the other stuff put together (topic for another post).

Today, I'm focusing on 2 - and sharing why I think that 2009 will go down as the year when 10G Ethernet takes off, and why VMware will be the thing that makes it happen.  Interested?   Read on.....