Virtual Geek

Understanding vSphere Disaster Recovery/Avoidance options – Part I

DEC 15th, 2010 – updated a bit based on feedback.

Interest from customers on EMC VPLEX is very, very hot, and that’s keeping us very, very busy.   Interest highlights that the core idea of active/active geographically dispersed transactional storage (“access anywhere”) is something people gravitate towards as they dream their next “active/active” virtualized datacenter dream.

The most common use case I see (admittedly, this will be biased towards things my team and I support) are stretched vSphere clusters.

The topic of geographically stretched vSphere clusters complex topic, and sometimes the customer doesn’t understand what they are asking for specifically, and sometimes we (the vendor community) don’t do a good job explaining the complexities involved.

Inter-site vMotion also makes for a SUPER COOL demo - and everyone loves a cool demo :-)   Couple that with the first paragraph of this post (the core idea resonating), and you have every sales person’s dream – customers who start reaching for their checkbooks…

Put that all together – stir in new technology and new ideas that people don’t fully understand, and you have a recipe for bad stuff to happen :-)

Ok – most of the stuff from my article WAY BACK WHEN (still worth a read – wow 2.5 years later) around geographically stretched clusters still holds true. 

I keep repeating myself (look in the QnA from this first big VPLEX post) – but it bears repeating: stretched vSphere clusters are possible, but you REALLY need to know what you’re doing.  

At VMworld Copenhagen, I met with the VMware High Availability team (responsible for VM HA, VDR, SRM) over dinner.   we discussed at length the confusion that exists in the market between disaster recovery, disaster avoidance, and the common refrain of “I could just script that”).  Then we moved on to a more productive discussion – realizing that we collectively (VMware + storage vendor community) weren’t helping by not being explicit enough about the solution pros/cons, as well as the longer term direction EMC is taking.

Here’s my first stab at trying to helping and clarify – but it’s a long topic, so I’ve broken it into 3 posts.

In Part I of this post: I’m going to try to summarize the 4 major categories of Disaster Avoidance (“I know the hurricane is coming”) and Disaster Recovery (“oh oh, the hurricane hit, and my datacenter is dead”) and do it in a way that applies to multiple vendor solutions in this space.

This is important – I’ve found that with customers, they mentally seem to just “blend these all together” (heck, people mentally blur VM HA and vMotion!), but they are VERY distinct choices.   Like all design choices - these 4 options always involve some trade off (doesn’t everything!).   If you’re interested, I’m going to outline them in the simplest way I can, and I’ll highlight the pros/cons,

In Part II of this post: I’m going to explain what you need to know IF you pick the stretched vSphere cluster using VM HA using EMC VPLEX, with a particular focus on the “partition” state.   Much of this will apply broadly to the category.  I don’t claim to be an authority on other vendors’ storage platforms – but many of the considerations are intrinsic to vSphere.

In Part III of this post: I’m going to explain what the next couple of years look like in this space (both product development and R&D).   As the VMware team and I discussed that dinner, we need to show people the path we’re working on if we want them to understand our advice today, tomorrow, and in the future.

It’s a nerdy (and long) journey – but if you’re in the market for vSphere in two or more sites, and looking for vMotion between sites AND disaster recovery, read on.   I’ll try my best to keep it focused – it’s not a “sound bite” topic.

Why I’m thankful….

(still plowing away on some heavyweight posts…. but it is American Thanksgiving, so….)

Most of all, I’m thankful for my family.

Next, I’m thankful for my friends.

..and I’m thankful to be lucky enough to be in the middle of what’s a very cool conflux of technologies, market changes, and to be in a gig that lets me lead a global team, and interact with customers and EMC, VMware and Cisco engineering in all sorts of cool ways.

It’s not all roses though - a peek into the after-hours life of vSpecialists…

We, like all our brothers and sisters within EMC and EMC partners work very hard.   We, like them – are driven by technology, customers, passion.   Sometimes, it takes a little extra coffee and Red Bull (EMC customer – thank you very much!) :-)

But – we also have fun :-)

So here’s the background….

1. After a full day of internal and customer meetings
2. After the vSpecialist manger GeekWeek.
   • BTW - GeekWeek is when new vSpecialists finish their onboarding, ending in a Kobayashi Maru style-challenge.  It’s an opportunity to play with/hands on/break with UCS.future, EMC.future, and VMware.future – trying to make things work that are crazy :-)   Suffice it to say that our team failed, but failed spectacularly :-)
   • We want the leaders (me included) to not lose our chops – so we go through it too – we all must know as much as we can. 
3. After a big VMUG in Atlanta
4. After a late night customer event post VMUG

Note - NO ONE KNOWS EVERYTHING, and we, like everyone, must always try to be humble.

After all that – we did something crazy/fun from 8:30pm to 1:30am that Wednesday.   I only point that out so no-one misunderstands and things we are not focused on the tasks at hand, or are being frivolous:

• our priority is our work – this stuff is “on top of the day job”
• this is the cheapest type of marketing possible, and great value for EMC’s dollar – guerilla marketing
• when you see it – we’re just having fun, don’t read into it any more than that.

And yes, it IS wicked awesome fun :-)  THANK YOU to my teammates.

If you’re interested, read on.   Here are preview shots as a tease.  You’ll see some of the output – which hopefully will amuse you (I GUARANTEE I’ll look like an idiot, but it’s all in good fun) – in December.

Now THIS is solid marketing…

As a nerd, I tend to have a bit of a knee-jerk when I see marketing (from EMC as much as anyone) – but I’ve come to realize:

1. It’s important – it’s a way to communicate messages clearly to the marketplace.   People who thinks success comes purely from engineering, sales, and support, well – I would disagree.   I think you need everything.
2. Done right it can be amusing or informative.
3. In today’s world, it should be interactive – a two way dialog.

Think of the now well-known Old Spice ads – they are a clear example of good marketing.   For people wondering if it’s good for business, I’ll tell you this – I know that without that ad campaign, I’m 100% sure I wouldn’t have used the words “Old Spice” in a virtual geek post :-)

So – this crossed my path, and I thought “cool”.   Click on the picture below if you want to give it a whirl.

As you answer the questions about your own journey to the private cloud, the person walks along through the phases of that journey, representing where commonly customers are based on those threshold events.

What’s really cool is that at the end, there’s a whole bunch of customer videos, quotes, documents (screenshot below) about what they’re doing in the stage where you are, and what they’re seeing as results.

I also REALLY dig that they don’t seem to be prompting for info to get solid content. I hate that.

 

Hats off to you EMC marketing – pretty swank!

Having trouble importing a big OVF/OVA?

Have been working on a couple long nerdy posts, so today’s is a short one…

Last week was the “Managers GeekWeek”, and one part of our Kobayashi Maru scenario involved doing some creative stuff with an “as of yet” not released into the wild EMC VSA.  

Importing the VSA kept failing for some reason, complaining about “not enough space on the datastore” vCenter error, when clearly the datastore was big enough.

Here’s me importing the VSA – it deploys fat right now as an engineering build – it’s 400GB when deployed on the datastore (which of course is thinly provisioned at the array level with VAAI, so it deploys super fast :-)

But – the datastore is 1.9TB in size, so why does it throw this error?

The answer to the question is the allocation size. 

You set this when you create the VMFS filesystem, and most people just click “next, next, finish”.   The default allocation size is 1MB, which correlates to a maximum single file of 256GB – and this is the maximum file in the filesystem, PERIOD (so any file, including any VMDK). In the example above, I was importing a virtual appliance which had a large, single VMDK (400GB).   The error message spells it out, but otherwise smart folks were missing it.

So – how do you resolve?   Easy.  You use a larger allocation size.  Since this is a destructive process on the datastore, if there are VMs on the datastore, use Storage vMotion to vacate the datastore, then delete the datastore and recreate it.   This dialog is where you set the VMFS allocation size.   Here, selecting the largest current allocation size (8MB) correlates with the largest filesize being 2048GB.

I haven’t seen a best practice on the topic, but it seems to me the downside of the larger allocation sizes (slightly less efficient for small files that are <8MB in size) seems to me to be a small one.   Every datastore I’ve every seen is dominated by a relatively small number of very large VDMKs (base VMDKs or snapshots and memory state snapshots) – so personally, the slight loss in efficiency I’ll take over having the ability to easily put in big appliances as I see fit. 

Personally, I’m just starting to default to the larger 4MB or 8MB size.   What say you?

Atlanta VMUG – EMC Keynote (VMware/Storage Best Practices)

Thanks everyone for attending the Atlanta VMUG – was great to see so many customers, friends, and colleagues.  THANK YOU.

A suggestion to all vendors: the EMC HoL was a HIT.   Customers loved to be able to actually TRY all the stuff we all talk about.  It’s obviously much harder to pull off than a little table with marketing whitepapers, but man, it’s a lot of value for the customers.  Here’s a shot before it even got rolling.

For the EMC keynote - the topic was "Next generation VMware/Storage best practices".  The intent was not to be an EMC commercial (I don’t think that’s the purpose of VMUGs) but rather focus on cross-vendor considerations, best practices, and how to detect and react to issues and considerations.

I got loads of requests for the presentation – so, here it is!

Like always, I’m posting in PPT format, so people can use how they see fit (click on the slide to download).  

Yup, that means that competitors and everyone can use/repurpose for positive and nefarious reasons.  Hey, I don’t worry about “the other guy” – I’m more focused on us doing the right thing for EMC customers, partners, VMware, EMC ourselves, and everyone I can think of.

Here’s a shaky-cam recording of the session (thanks Scott!)

Atlanta-VMUG-EMC Keynote from Chad Sakac on Vimeo.

One of the global faces of VCE :-)

The key thing that often gets lost in the blogosphere/twitter world is that the people are people, plain and simple.   In the end, it’s about the humans, the relationships between humans – at least as much as having a joint vision and kick-a$$ technology and products (which we’ve got in spades :-)

The team in Sweden sent this pic from a VCE customer event this week…

In the picture – it’s a human face on VCE: Magnus Bergman @ VMware, Greger Jerlehagen @ Cisco and Jonas Jansson @ EMC – and here’s the back of the shirt :-)

A little fun goes a long way :-)

Wish I could have been there guys!

Come to the Altanta VMUG tomorrow!

Setup underway… Hope to see you there!   I’m presenting from 10:30-11:30, topic will be “Next Generation Best Practices for Storage and VMware”.   Working on it now to make it as vendor-neutral as possible, as I know what VMUG folks want/don’t want to hear (hint, they don’t want to hear a vendor commercial)

Details (logistics/agenda) here: http://communities.vmware.com/community/vmug/forums/us-southeast/atlanta

Topics covered:

• How have best practices evolved, and where do we stand in 2010/2011?
• Every rule has an exception – what are the exceptions to the best practices?
• How do you analyze and get yourself out of jams?
• What’s coming?

Updated VPLEX HA KB article (re vSphere 4.1)

FYI – the VPLEX team and the vSphere team have updated the VMware KB article that refers to VPLEX behavior in stretched vSphere cluster use cases to include vSphere 4.1

To recap – this use case (formalized test harness for stretched vSphere clusters) is a relatively new thing.   Unlike the formal VMware HCL process for storage devices, which is mature, this is an area where VMware and the storage vendors that support this sort of thing are treading on new ground (i.e. formal testing the myriad of failure cases).

We published this back with vSphere 4.0 support, and just finished the explicit vSphere 4.1 testing work, and VMware just finished reviewing the results.

You can check it out here.

BTW – Host Affinity is not noted, not because it wouldn’t work in this use case – but because that test harness is not finalized, and all the various implications (in various partition behaviors at the VPLEX and vSphere layers) isn’t quantified.

View 4.5 via NFS/FAST Cache on Vblock 1

Noticed last night that this whitepaper got posted.  

It shows a series of workloads, the effect of FAST Cache on the NFS performance.   It used 10GbE end to end and VPC configurations.   It shows the effect of different client/core scaling (from 96-155 per B200-M2 @ 96GB RAM), commentary about newer blades, along with the details on the storage and the network.

There’s a high level doc (click below) and a detailed one (also below).   One thing that has been a good shift is that the VCE team has made it clear to everyone – ALL SOLUTION DOCS, ALL PUBLIC, ALL THE TIME (posted at www.vce.com) .   Makes it easier for everyone :-)   There are great ones on SAP and other enterprise apps, and also some customer case studies.

On a less sexy, but very practical note - for VCE-certified partners – the new Bill-of-Materials tool (simplifies flexible configuration, quoting) is out.  

It’s not perfect, but my team’s feedback is that it’s a great step forward, and is being continuously improved.   Strongly suggest reaching out to VCE to get trained (500+ people at VCE partners trained already), as using this tool simplifies and accelerates things.   It will constantly be updated to link up to the Vblock configurations rolling off the two manufacturing lines.

Ok – back to the sexy “solutions on Vblock” docs – read and have fun!

 

Why people who say the following are wrong…

It’s always interesting to watch vendors (EMC included) make the “you don’t really need _____” when they don’t have a given feature, while all the time, they are furiously working on it.

Likewise, it’s always frustrating (to everyone) when a customer expresses requirements in the form of a feature/implementation, rather than a broader requirement (ergo they’ve fallen for a vendor’s “cheat sheet” selling motion).

This post was triggered for me by two things that happened last week: 1) watching the SNIA twitter feed from @jpwarren (disclosure I wasn’t there); and 2) some conversations at the EMC TC conference that happened in Paris.  

First… the SNIA conversation seemed to be about a multi-vendor panel (@Rodos, @jpwarren, you were there, perhaps you can comment): 

• Vendor A seemed to be all about automation, virtualization of 3rd party storage, and autotiering.  When pressed on the topic of dedupe, they talk about dedupe on their NAS heads, but states that the bulk of dedupe needs are in the backup storage market (where inline is a pre-requisite), not primary storage.
• Vendor B seemed to be all about all about dedupe of ALL storage being the answer, convergence of protocols, vCenter plugins, that mega-caches and primary storage dedupe are all you need - that automated tiering is rubbish.

I’m wondering if readers can guess who Vendor A and Vendor B are :-)   They weren’t EMC – but I simply didn’t see EMC in @jpwarren’s twitter feed (though he did a blog post subsequently).   Do you think either of them are flat out wrong/right – or maybe they are both kinda right/wrong at the same time?

Second… At the TC conference, I asked a room full of TCs if dedupe of primary storage was, in their eyes, a good or a bad thing (simple question which glosses over a ton of “it depends” variables).   On the whole, they responded that primary storage dedupe is a BAD thing.   I personally disagree.  Of course, the “it depends” is dependent on a ton of factors, but there’s nothing intrinsically wrong with primary storage dedupe.

BTW – this applies to ANY industry, not just storage land.   I would also bet that Vendor A is furiously working on primary storage dedupe, and I would be that Vendor B is furiously working on automated tiering.

If you’re interested in a little more examples, including customer perspectives, read on…