Virtual Geek

A “Multivendor Post” to help our mutual iSCSI customers using VMware

UPDATE: Sept 21st, 2009 – 4:24pm EST – Are you using vSphere, or thinking about it?   We’ve done an updated “Multivendor iSCSI post” with vSphere details here.  If using ESX 3.x read on…

Today’s post is one you don’t often find in the blogosphere, see today’s post is a collaborative effort initiated by me, Chad Sakac (EMC), which includes contributions from Andy Banta (VMware), Vaughn Stewart (NetApp), Eric Schott (Dell/EqualLogic), and Adam Carter (HP/Lefthand), David Black (EMC) and various other folks at each of the companies.

Together, our companies make up the large majority of the iSCSI market, all make great iSCSI targets, and we (as individuals and companies) all want our customers to have iSCSI success.

I have to say, I see this one often - customer struggling to get high throughput out of iSCSI targets on ESX.   Sometimes they are OK with that, but often I hear this comment: "…My internal SAS controller can drive 4-5x the throughput of an iSCSI LUN…"   

Can you get high throughput with iSCSI with GbE on ESX?   The answer is YES.  But there are some complications, and some configuration steps that are not immediately apparent. You need to understanding some iSCSI fundamentals, some Link Aggregation fundamentals, and know some ESX internals – none of which are immediately obvious…

If you’re interested (and who wouldn’t be interested with a great topic and a bizzaro-world “multi-vendor collaboration”... I can feel the space-time continuum collapsing around me :-), read on...

Are you a EMC ControlCenter and VMware customer? Good hotfix to know about....

Was working a customer case where they were having issues with VMotion and Control Center.  Control Center is one of the only Storage Resource Management tools with tons of VMware integration - and has had it for more than a year.

Ok, Control Center has active discovery (i.e. agent) and passive discovery mechanisms (i.e. leverage an API).   Most VMware-integrated features use the passive mechanism - basically ControlCenter has a VM-proxy (i.e. a host which can be a VM that has software that queries the ESX CIM API and the VC APIs).   Some customers want more.  For example, they want file-level reporting for chargeback and analysis WITHIN the VM.  Or, they want drill-downs into databases in VMs.   These require agents to run inside the VM if you want that (again, ESX-layer chargeback, utilization reports, thin provisioning use, performance and VMDK->Datastore->LUN/filesystem->backend details - those have no agent requirement).

So, the problem only arises when ControlCenter components running on a guest systems (ergo agents to provide file-level reporting or application integration) are Vmotioned and ControlCenter has discovered the ESX server hosting the VMs that those ControlCenter components are running on.

The root cause is that the ControlCenter store component incorrectly interprets a Vmotion activity as a deletion of a VM (i.e. at the ESX CIM level, the VM just disappeared, and requires correlation at the VC layer).  That deletion is not allowed by other rules within the ControlCenter repository and the discovery of the ESX server fails resulting in stale data in ControlCenter and sometimes corruption to ControlCenter component configuration data.

We just released a hotfix for this problem for EMC Control Center 6.1 (HF4486) and will be releasing one for 6.0 shortly (HF4485).

This primus case contains more information regarding the problem.  emc203588

http://knowledgebase.emc.com/emcice/documentDisplay.do;jsessionid=0735EE24ACC9E5D50A82A6A5D8CF9820?docType=1006&clusterName=DefaultCluster&resultType=5002&groupId=1&page=&docProp=$solution_id&docPropValue=emc203588&passedTitle=null

Based on our testing, once the correct hotfix is applied, Vmotion of VMs with ControlCenter components installed will work fine and there will be no support limitations.

Thanks ControlCenter team!!

"Physical servers kill baby penguins... go virtual"

ROTFL!  

Clearly not the work of a marketing genius, but this one had me smiling.   I can't take credit - was EMC's Derrick Baxter - the guy who heads up a huge chunk of our Exchange on VMware testing here at EMC.   He was part of the original Exchange Ranger (now called Microsoft Certified Master program) rotation when he was at Microsoft, so to call him "a vet" is being woefully understated.  They also trained those folks to be hyper-conservative - ultra "do it this way and it works".     I'll never forget when VMware and EMC started the first test - he was.... let's say... skeptical.

Now the title of this post is his hotmail sig.   The power of virtualization with VMware has made another convert :-)

Oh - BTW - if you missed it, we just added to the Joint EMC/VMware Reference Architecture library (this one done by the OTHER crack Exchange squad at EMC in Cork Ireland)...

"Metropolitan Exchange Recovery in a VMware Environment" - H5805, available on Powerlink.

We have solutions where we have protected via CCR (because sometimes that's the right way), via SCC - like here (because sometimes that's the right way), and via SRM (because often that's the right way).

Oh, while I'm at it, I've said it before, I'll say it again... you've got to love laughing... and today I got two doozies...   The counterpoint to VMware's vExpert program, Stu's vIdiot program :-)

VMware and Documentum eRoom - or "how EMC saved $1.5M"

I've noted before how EMC uses VMware widely - there are more than 3000 ESX servers in use in various places, so it's not uncommon to hear great stories about how we save money.  This one was a nice one.

The full EMC Documentum suite works great on VMware (you only can't virtualize the FAST index server - and that's because of a weird Microsoft thing that's getting resolved, but that's only one server of many).   EMC Documentum is really a broad set of stuff, one part of which is called eRoom, which is analagous to how most people use Sharepoint - document storage and collaboration.   FYI, we use both Sharepoint and EMC Documentum internally.

I was happy to hear we run the whole deal on VMware Infrastructure.   Read the write up if you want the details, but here are some highlights...

1) the usual CapEx reasons (more important these days than ever!)

"Eliminated additional $1.5 milliion from planned budget expenditures and reigned in ongoing hardware and associated costs"

2) the normal "hey look at all this unexpected operational flexibility upside" reasons

"...several fail-overs have already been handled by VMware - the entire process only took a few minutes and the eRoom users never noticed a disruption.   In the past, such a failover would have significantly impacted users and produced longer down times."

3) the always fun "wow - performance is good.   WOW - really good!".    Everyone always is worried about this, almost always incorrectly.   Higher utilization can also be expressed as higher efficiency, which for some workloads has an effect of getting MORE on the same hardware (i.e. in this case we could have more front-ends).

"the final performance results acheived were significantly greater than expected.  Users were both pleased and astounded with the increased performancel eRoom activities such as site access, opening stored content, and saving files, are now instantaneous.   Response delays have been eliminated, yet the virtual site configuration is nearly identical to the original physical hardware configuration."

In the end, this one says it all....

 

So... You aren't virtualizing application ______   because of.... ?

What does this mean? "EMC Becomes a VMware Authorized Consultant (VAC)"

Welcome to 2009!

We've been working on this for a long time - and it's just the beginning, with lots of work through 2009.   If you didn't see it - here's the PR:

http://biz.yahoo.com/prnews/090106/ne55718.html?.v=1

Here was my pre-new year hint at this - and I think impressive (the graphs continue that way through the new year).   We've been building this for a while.

I bounce between nerd posts and business posts (my job has a foot planted in both worlds).   Read on for a bit of unofficial (but from the heart) questions and my answers.

Updated "Homebrew" ESX Hardware List

One of the most popular 2008 articles I've posted was this one on how to build a home-brew ESX server.   Within EMC it's a game now for our nerd elite to come up with creative ways to do it cheaper, faster, smaller, and cooler :-)

Keep reading for the new (at least for now!) record holder, from Adam Jones a Sr. TC, VCP (and more acronyms behind his sig than I've ever seen :-) in Kansas City with a dual core, x86-64 (ergo future ready), 8GB machine....

Do you have your own home-brew story?   I'd love to hear it!

Fantastic esxtop post on VMware communities

If you're like me, you spend a fair amount of time looking at esxtop data.   There's a really great post that just went up on the VMware communities I think is just fantastic if you are looking to interpret results and wanted to share!

http://communities.vmware.com/docs/DOC-9279 

BTW - Happy new year to all - and here's hoping that 2009 is filled with peace, happiness, surprises, and interesting challenges for us all!

Scads of posts I've been contemplating over the holiday, chomping to get back in business!