Virtual Geek

A “Multivendor Post” to help our mutual iSCSI customers using VMware

UPDATE: Sept 21st, 2009 – 4:24pm EST – Are you using vSphere, or thinking about it?   We’ve done an updated “Multivendor iSCSI post” with vSphere details here.  If using ESX 3.x read on…

Today’s post is one you don’t often find in the blogosphere, see today’s post is a collaborative effort initiated by me, Chad Sakac (EMC), which includes contributions from Andy Banta (VMware), Vaughn Stewart (NetApp), Eric Schott (Dell/EqualLogic), and Adam Carter (HP/Lefthand), David Black (EMC) and various other folks at each of the companies.

Together, our companies make up the large majority of the iSCSI market, all make great iSCSI targets, and we (as individuals and companies) all want our customers to have iSCSI success.

I have to say, I see this one often - customer struggling to get high throughput out of iSCSI targets on ESX.   Sometimes they are OK with that, but often I hear this comment: "…My internal SAS controller can drive 4-5x the throughput of an iSCSI LUN…"   

Can you get high throughput with iSCSI with GbE on ESX?   The answer is YES.  But there are some complications, and some configuration steps that are not immediately apparent. You need to understanding some iSCSI fundamentals, some Link Aggregation fundamentals, and know some ESX internals – none of which are immediately obvious…

If you’re interested (and who wouldn’t be interested with a great topic and a bizzaro-world “multi-vendor collaboration”... I can feel the space-time continuum collapsing around me :-), read on...

Are you a EMC ControlCenter and VMware customer? Good hotfix to know about....

Was working a customer case where they were having issues with VMotion and Control Center.  Control Center is one of the only Storage Resource Management tools with tons of VMware integration - and has had it for more than a year.

Ok, Control Center has active discovery (i.e. agent) and passive discovery mechanisms (i.e. leverage an API).   Most VMware-integrated features use the passive mechanism - basically ControlCenter has a VM-proxy (i.e. a host which can be a VM that has software that queries the ESX CIM API and the VC APIs).   Some customers want more.  For example, they want file-level reporting for chargeback and analysis WITHIN the VM.  Or, they want drill-downs into databases in VMs.   These require agents to run inside the VM if you want that (again, ESX-layer chargeback, utilization reports, thin provisioning use, performance and VMDK->Datastore->LUN/filesystem->backend details - those have no agent requirement).

So, the problem only arises when ControlCenter components running on a guest systems (ergo agents to provide file-level reporting or application integration) are Vmotioned and ControlCenter has discovered the ESX server hosting the VMs that those ControlCenter components are running on.

The root cause is that the ControlCenter store component incorrectly interprets a Vmotion activity as a deletion of a VM (i.e. at the ESX CIM level, the VM just disappeared, and requires correlation at the VC layer).  That deletion is not allowed by other rules within the ControlCenter repository and the discovery of the ESX server fails resulting in stale data in ControlCenter and sometimes corruption to ControlCenter component configuration data.

We just released a hotfix for this problem for EMC Control Center 6.1 (HF4486) and will be releasing one for 6.0 shortly (HF4485).

This primus case contains more information regarding the problem.  emc203588

http://knowledgebase.emc.com/emcice/documentDisplay.do;jsessionid=0735EE24ACC9E5D50A82A6A5D8CF9820?docType=1006&clusterName=DefaultCluster&resultType=5002&groupId=1&page=&docProp=$solution_id&docPropValue=emc203588&passedTitle=null

Based on our testing, once the correct hotfix is applied, Vmotion of VMs with ControlCenter components installed will work fine and there will be no support limitations.

Thanks ControlCenter team!!

Updated "Homebrew" ESX Hardware List

One of the most popular 2008 articles I've posted was this one on how to build a home-brew ESX server.   Within EMC it's a game now for our nerd elite to come up with creative ways to do it cheaper, faster, smaller, and cooler :-)

Keep reading for the new (at least for now!) record holder, from Adam Jones a Sr. TC, VCP (and more acronyms behind his sig than I've ever seen :-) in Kansas City with a dual core, x86-64 (ergo future ready), 8GB machine....

Do you have your own home-brew story?   I'd love to hear it!

"Site Recovery Manager in a Can" - Celerra Virtual Appliance HOWTO 401

UPDATED Jan 19th, 2009 - the updated doc (v1.01) reflects SRM update 1, general minor corrections, and the removal of the ONLY step (Solutions Enabler) that required an EMC powerlink login.

Folks, here it is - the "all-in" step-by-step guide for setting up Site Recovery Manager with nothing more than two ESX servers (which you can even do with home-brew servers, as noted here) and the Celerra VSA.

If this is your first read, you can either just click on the document below or start the 101, 201, 301, 401 journey here.

This guide isn't the first one of these out there.  I think my respected friend Adam Carter at Lefthand did it first (Adam, hope you're enjoying HP!).  The NetApp one is pretty good too.   What I like about ours is the extreme completeness.   It's also nice that the Celerra VSA has no timeouts, no limits (except that you'e not allowed to use it in production).

A quick shout out...   This was absolutely yeoman's work by one of the newest members of the Global EMC VMware Specialist squad - Bernie Baker, with strong assists from one of the other newbies - Stephen Spellicy.   Both of these guys started relatively recently, and this work is (IMHO) FANTASTIC - it highlights that this stuff can be EASY (and that they have what it takes to have a early impact).  Bernie - you've set the bar high for the team and yourself - GREAT WORK!

Until the next major revision of the Celerra VSA (which will be linked to major feature releases), expect to see only one more "301"-level Celerra VSA post (downloading and using EMC Replication Manager for VMware for point-in-time, VM-consistent instant VM backup/restore) - we're warming up in the batter's box a series on Avamar Virtual Edition....

Virtualization Security - Cage Match... FIGHT!

Now this is something I want to see.......

Today, RSA integrates with VMware in an couple of what I would call "useful but not earth-shattering" points - you can integrate envision authentication with Virtual Center and it also integrates with VDM 2.1 and VMware View Manager for hardened authentication.

But, with VMware as mission-critical as it is, security focus is getting strong.    At our recent Quarterly Technology Review, there were LOTS of really cool ideas where to go next, and ideas I would consider earth-shattering, but are so fuzzy at this point, there's no point in going on about it.   Let us do some more work, and then I'll talk about it again.

Now - The RSA conference is pretty cool, and this is an example of security moving to the forefront.

"In this verbal cage match session, two well known critics of virtualization security take on two virtualization company CTOs as they spar over how best to secure virtualization platforms: who should be responsible for securing it, and how that ultimately impacts customers and attackers.  We have Hoff and Skoudis versus Crosby and Herrod.  Refereeing will be respected analyst, Antonopoulos."

http://rationalsecurity.typepad.com/blog/2008/12/beyond-the-sumo-match-crosby-herrod-skoudis-and-hoffvirtsec-death-match-rsa.html

VMware View Manager/Composer 1000 Client Reference Architecture

A shout out to my compatriots on the VMware View team, and congrats on the VMware View launch two weeks ago... Congrats!   I've been heads down with our Quarterly Business Review and Quarterly Technical Reviews at VMware this past week, so a bit delayed on this post.....

We've been slaving away on a lower-scale VMware View building block, something more broadly useful for the last couple of months at the Santa Clara Lab I mentioned here: ; and where we have done 10K client scaling testing and 250 user building block VDM 2.1 reference architectures I mentioned here.  

BTW - If YOU are looking at a large scale (let's say 5K clients or more) Client Virtualization project, reach out to your EMC and VMware account teams and ask for some time with the engineering teams in Santa Clara - we built it to openly share our learning.....

Back to the subject at hand... So, I'm happy to say you can get it now - a VMware View Manager/VMware View Composer 1000 client Building Block Reference Architecture.   Read on for more!....

Real World Experiences Using Storage VMotion

One of the most disruptive events for customers is often a storage platform upgrade, since everything is consolidated - ergo everything is potentially affected.   EMC has a multitude of ways to do this... lemme think... right now I can think of 4 (non-disruptive platform upgrade in many CX cases, PowerPath Migration Enabler, Invista/Rainfinity, and OpenMigrator/SANCopy).   Every one has a specific use case (every customer is different).    But  a new tool in the toolbox is making a strong entrance on the migration scene for customers using any VI 3.5 release - Storage VMotion. 

As you'd imagine, EMC does a lot of these data migrations, and recently there was a big internal thread on this I thought it would be good to share, as well as sharing customer experiences.... 

Storage VMotion is obviously only useful when the data is VMware, and (for now) iSCSI and FC only.   It works generally with NFS datastores , it's not supported - I've seen some flaky cases so I don't think VMware is being arbitrary.   NFS as a source and a target is expected to be officially supported in the future.   

That caveat list is pretty short.   If it is VMware data (one more great reason to Virtualize?!), svmotion can be a fantastic option, and joins the other mechanisms as a key tool.

Read on for a couple of my svmotion "rules of thumb" and a couple customer examples....

EMC, Storage Resource Management, ControlCenter, VMware and Corporate Culture

I don't think company culture comes from the top down - or from the bottom up.   I think it comes from what people act and what they try to do every day - at every level.   I think people overestimate the ability of the people at the top to cause large-scale change (though it's certainly proportionately higher, and a great litmus test of leadership skill), and I think people underestimate the ability for individuals at all other levels to have an impact.

What's the connection to the post title?   EMC ControlCenter is Storage Resource Management software - and also integrates with the VC APIs and ESX CIM APIs for an end to end view.   There's  a demo here.

So - we have a customer who isn't happy with the state of Storage Resource Management software in general, and ControlCenter quite specifically here.  Ouch.   But, customer is always right (though StorageBod - I have heard that customers are much happier with 6.x than 5.x - but hold us to that).

What I really like is this response from the ControlCenter team - to me transparency is good, and facing the challenge is good.  Specifically - I think it's EXACTLY an example of the opportunity to have an impact.   Storage Resource Management is a challenge for our largest customers - and here you have someone taking the critique, and offering to take MORE - to take that customer input, and make the product the best it can be.  

I run a ControlCenter VM, and Dave, I'll take you up on your challenge.   Get that community open in January.   BTW - our EMC/VMware Open Community Wiki should be up early in the year also....

Does VMware DPM shorten ESX server lifespan?

The inimitable MikeD (http://www.mikedipetrillo.com/mikedvirtualization/) pinged me and some of my colleagues this week with this question - he was starting to hear it, thought it sounded like FUD.   The question is:

"Does the Distributed Power Management power-on/off cycle cause the boot LUNs of the ESX servers to fail prematurely?"

BTW - if you're not a DPM expert, this video highlights it's incredible promise (it's currently an experimental feature in VI 3.5).

The answer in a word?  NO.    DPM does not shorten ESX server's lifespan due to drive failure.

Ok - two comments.   The first is that FUD is always funny (sometimes even our own - and yes, I know sometimes EMC is the FUDer not just the FUDee - though I try as hard as I can to not do that).   The second is that there is more than just the one word answer of course, otherwise it wouldn't be another long-winded Virtualgeek post, would it? :-)

If you're interested.... Read on!

Version 4 of the PowerVDI tool

This tool continues to get updated (thank you Jeff Purcell who did SO much on this and the previous release, Dan Baskette and the Celerra team!).   You can get the latest version here:

http://powervdi.vcloud9.net/

This version is much faster - at least 2-3 times faster than the old one.   In terms of speed, I think it would be hard to get faster than this - in our "playing around", it's at least 10x times faster than the upcoming View Composer approach.

So - does that mean we think the array is the place to do this rather than the ESX server (aka VMware View Composer) long term or in all cases? NO. 

There are so many things that can be done re: image preparation and handling and management better at the VMware layer than the storage array layer - which are both as important as speed.   All the approaches have similar cost savings.

We're embracing the upcoming stuff (stay tuned - another blog post coming on this in a couple weeks).   Over time, we're going to work to merge them as much as possible (i.e. use the VMware View tools to manage, but where speed of mass distribution is the key - you can choose to do it at the array layer rather than at the VMware layer (but it's managed at the VMware layer).