State of the art – integrating storage and VMware – Part I

In my opinion – the ideal infrastructure is one that is literally invisible.   It would respond fluidly and dynamically to fluid requirements at the application level – tuning capacity, performance, cost, and availability based on higher level SLAs.  It would do this non-disruptively, and it would do it with a high degree of abstraction.

To be honest, we’re not there yet.  

Managing storage with VMware is complex (EMC as much as every vendor) and making it all work together requires that you have detailed understanding of both the application and the infrastructure.   Long and short, we’re burdening poor humans to bridge the gap – demanding that VMware admins become more expert on storage and storage admins more expert on VMware.   We demand process and paperwork to glue the bits together.

far from invisible…

We’ve got a long way to go – but we are diligently working down that path.   Regarding storage, vSphere has the first baby steps (improved Storage Views and managed datastore objects), and the vStorage APIs for Array Integation (VAAI) will continue to take us down this path in the next minor and next major vSphere series releases.   I think people would we blown away how far out we’re working on this stuff.   Sure, after 2011 it gets VERY squishy, but we’re working on it, and it’s VERY cool!

Now – while that’s interesting, what have we done for you lately?  

In the near term, we’re trying to cut through it by making storage visible in the natural language of both the VMware administrator, and virtual machine visible in the natural language of the Storage administrator.  

to put this in perspective, here are some of the results from the survey I posted earlier here (always happy to have more data!). 

Fascinating to see the detailed responses from folks – and while I’m happy to be a proud EMCer (while proud, we all have much to improve!) , it’s great to see so many folks using all sorts of stuff (EMC and otherwise) find the blog useful!

This is split into two parts:

• Part I – making storage visible in the natural language of the VMware administrator (new tools, as well as exposing our thinking on this topic)
• Part II - making virtual machines visible in the natural language of the storage administrator (a sneak peek at something right around the corner)

The topic was long enough, and needed some  and includes details and HOWTOs for anyone using VMware and EMC storage – hence breaking it into two parts.

If you use VMware and shared storage – particularly if you use EMC, you should find this INTERESTNG!

Read on…

Without further ado… part I.

The natural language/natural GUI for a VMware administrator is vCenter.  So – for you VMware admins (187 of you responded above), how often would it have been useful:

• to be able to quickly figure out which datastores are actually which devices on the arrays for troubleshooting?
• to - in a single screen – see the relationship between a VM, it’s virtual disks, the datastores on which they reside, and the actual array objects?
• to have been able to quickly make sure that core array configuration best practices are implemented including target configuration?
• to see the actual backend configuration, including virtual provisioning, multipathing config, protection and remote replication levels?

To give VMware admins visibility into EMC storage we started with EMC Storage Viewer.   This is a vCenter plugin that provides details at the datastore, storage object, virtual disk, and target-level configuration detail by integrating EMC storage APIs (Symmetrix and CLARiiON) with the VI SDK.  

It’s simple, and free.

We introduced version 1.x as a not-formally supported utility for VI3.5 at VMworld 2009 Europe – and customers LOVED IT.    We have now GA’ed version 2.0 which supports vSphere 4.   This is generally available, and supported by EMC (but is still simple, and still free!).   It has very broad and deep integration with EMC CLARiiON and EMC Symmetrix arrays, and I would consider it a critical tool for VMware and EMC customers.

Here’s the video to give you an idea of what it does.   While this video of version 1.1 using VI3.5 and a Symm, the functionality in the vCenter 4 plugin is very similar (better, but similar), and does more in the CLARiiON use case.   It also works with blended block/NFS use cases.

you can download the high-rez version of this video here.

Here’s a HOWTO video (where to get it, how to install it):

you can download the high-rez version here.

A couple of caveats – it:

• because of changes in the vCenter plugin model between Virtual Center 2.5 and vCenter 4, EMC Storage Viewer v2.0 only works with vCenter 4 (and not Virtual Center 2.5).   You can still use the earlier EMC Storage Viewer 1.1 – though note that it is not formally supported.
• It currently throws a non-critical error in CLARiiON-only environments on one screen (one query expects CLARiiON and Symmetrix) – this is getting fixed in the next revision.
• If you are using a version of FLARE later than FLARE 28 (the current GA latest version) you need Solutions Enabler 7.0.1
• While it coexists with NFS, there is limited detail shown for Celerra NFS and iSCSI support is limited – working on it.

There’s more to come near term.    Here’s what’s coming, but I want to know - what would you like above and beyond this list?

• better NFS support
• More detail including more performance data
• Simpler configuration in CLARiiON-only configurations (minimizing the solutions enabler steps)

And, remember, this is our near term answer to the first half of “making infrastructure more invisible” for VMware administrators.  Ideally, the vApp container would define the policy, and the infrastructure would deliver as best as possible against that SLA – making these “integrated views” in the long term… irrelevant.

If you like this though…wait till you see part II– making Virtual Machine visible in the natural language of the Storage administrator!

Comments

[David Barker]

Ok, it's taken a while, but here's a quick idea...

Most of the integration so far seems to be 'passive' - aimed at viewing existing info. That's fine, but I'd like to script with the info too.

e.g. If you're collecting backend performance info, I'd like to schedule stuff around it.
- Kick off/pause VCB backups.
- Zero dirty blocks in vmdk's during periods of low I/O. Its possible to script MS's 'sdelete' (see http://www.yellow-bricks.com/2008/01/04/vmware-consolidated-backup-and-deleted-files/), but it doesn't know what its doing to backend performance .

[Chad Sakac]

Thanks David - appreciate the feedback. We're treading carefully around the "active" management piece - we're getting mixed feedback.

Our customers that have storage admins and vmware admins find the topic radioactive - they love the idea that each can VIEW the info they need across domains, but not affect it (some exceptions seem "kosher" with that community - like replicas).

Conversely, smaller shops that have one person doing both want it

I don't think one or the other are right/wrong - just different.

For mid-to-large shops, if you look at part II of this - you can see that we're providing the same visibility, but in the other direction as well - in the native context where people in the shops where there are storage admins can provision and take action. This is delivered by integrating those directly with the vCenter APIs.

For smaller shops, you will see more on this topic at VMworld this year, as more plugins are announced/discussed.

Performance info is coming soon. Good suggestions on zeroing and VCB/vStorage API for data protection

[Fabio]

In the first video I noticed the long list of luns: each of them has 32GB of size.
There is a technical reason to provide a such long list of little luns or it was only to fill the screen, for the video purpose?
Thanks