Virtual Geek

Insider’s Perspective: Ionix and VMware

The sub-title of my blog is “an insider's perspective, technical tips n' tricks in the era of the VMware Revolution”.  I’ve said it before, I’ll say it again – I think I’ve got the coolest job on the planet :-)   I’ll give you a couple “insider” hints.   2010 will be a very big year on the VMware front – loads of big releases, new functionality and expanding use cases.   Likewise, it’s shaping up to be a block-buster year on the product front at EMC.   The number of “headline” level things down the storage and security fronts between now and EMC World (May) are huge.   Between now and the end of the year – massive.

So today, a big move occurred on the management side of the house that’s been underway for a while.   You can read more here. For more detail, and that “insider perspective”, read on…

Solving a weird “slow performance” cloning issue…

Recently was working a customer case with my VMware colleagues where a customer was seeing that cloning operations were taking a lot longer on their V-Max than it was on their mid-range CXes.   Turned out to be a tricky case, and taught me something new.

This experience would apply across more than just EMC arrays, so I thought I would share what we found, how we found it, and what we did about it – in the hopes of helping folks our there.

If you’re interested – read on….

Interested in a 40K employee company’s experience with View 4 deployment?

These EMC IT webcasts are fantastic, and customer feedback has been very positive.   They break down the good/bad/ugly of our journey to the private cloud.

This week’s session (Thursday, 8am PT/11am ET)  is on EMC’s own deployment of View 4.  

I’m using it right now, and love it!  Of course, there’s still a long way to go (check-in/check-out is big for me).   I use it all the time to access my lab on the corp network – fire up the View client and I am in the core network (extra secured via RSA, which integrates with View Manager).

We’re still in the pilot stages with our View 4/Windows 7 activities (hundreds of clients), but so are a lot of customers.   Join in for a perspective (and the upside of the EMC IT folks presenting themselves is it’s not a salesperson :-)  on a 40K employee company’s deployment of client virtualization.

http://info.emc.com/mk/get/DBM6065-3450_raf_lp

BTW – this is part of a large series.   EMC IT is a massive consumer of the technologies we develop, and is rapidly approaching the 100% virtualized mark for our x86 workloads.

They are bravely (warts and all) are describing that journey to the private Cloud that Joe and Paul (and I) believe in.   They are describing it across all use cases publicly.   You can follow that here:

http://emcvirtualizationjourney.wordpress.com/

I’ve asked them to keep it a “marketing free zone” – please check it out and let me know what you think!

My evil doppleganger

In all seriousness – there’s a ton of going on at EMC focused on Hyper-V, and while I don’t want to help him too much, I do want to make the introduction :-)

I mentioned him earlier – Adrian Simays.   Adrian is “Mr Hyper-V” at EMC – and has now been gathering his strength to emerge publicly.   You can read up on EMC stuff focused on Hyper-V here:  Virtual Winfrastructure

There’s a great post on all the technical docs on how to leverage Hyper-V on EMC kit which I’m grabbing the links.

• EMC Symmetrix with Microsoft Hyper-V Virtualization
• EMC CLARiiON with Microsoft Hyper-V Server
• Best Practices: Using EMC Celerra IP Storage with Microsoft Hyper-V R2 over iSCSI
• EMC Replication Manager for Virtualized Environments
• EMC Blueprint:  Backup and Recovery for Microsoft Sharepoint Server Enabled by EMC CX, Replication Manager and Microsoft Hyper-V
• EMC Backup and Recovery for Microsoft Sharepoint Server with Hyper-V
• EMC Networker: Complete Protection for Microsoft Hyper-V
• Proven Solution: EMC Backup and Recovery for Microsoft Exchange 2007 Enabled by EMC CX, Replication Manager and Microsoft Hyper-V using iSCSI
• Reference Architecture: EMC Backup and Recovery for Microsoft Exchange 2007 Enabled by EMC CX, Replication Manager and Microsoft Hyper-V using iSCSI
• Reference Architecture: EMC Solutions for Microsoft SQL Server on EMC Unified Storage Platforms
• Reference Architecture: EMC Virtual Architecture for Microsoft Sharepoint Server 2007 Enabled by EMC CX and Hyper-V
• Disaster Recovery in a Geographically Dispersed Cross-Site Virtual Environment Enabled by EMC CX, RecoverPoint and Microsoft Hyper-V

BTW – if you’re a Microsoft IT user interested at the intersection with infrastructure, another great technical blog to follow (IMO) is Brian’s Power Windows Blog.   Brian, for an example recently posted on all the cool sessions at EMC World that touch on Microsoft technologies.

Now, of course, we have MORE of all of this on VMware – a lot more!   It’s just proportionally more based on how often our customers ask for VMware-specific content and integration rather than Hyper-V, but now I’m just smack-talkin’…   Adrian, it’s on :-)

 

Want to ask an EMC vSpecialist a question? (or… how to get a free ix2/ix4)

One thing we’ve been getting a LOT of requests for is a more public, consolidated place to get information about EMC/VMware stuff, ask technical questions and get technical answers, and where to find out what’s going on.

Inside the walls of EMC, we are using a social media portal (one.emc.com) which is great – an interactive place that is open, and easily searchable.    Externally, however, not so much.   EMC.com is fine, but not interactive.   Powerlink works, but since search is less than perfect – coupled with once again a "non interactive” sense, well…

So… I’m happy to announce the public availability of the “Everything VMware at EMC” public community!   This is an awesome site, and it’s totally open – to the public, to partners, heck, to competitors.

you can find it at the easy to remember URL:

www.emc.com/vmwarecommunity

So – what’s there?

• Discussion forums.   These are monitored by the vSpecialists – a great cadre of EMC, VMware, and Cisco technical experts.   They are all unbelievably skilled.   You can ask anything you want, and we’ll do our best to answer.   Just remember – this is not a replacement for support, as it is best effort.  If you have a support case, immediately call 1-800-svc-4-EMC or VMware.    Also, it is focused on the intersection of EMC and VMware – pure VMware stuff, please go to the VMTN community.  Subscribe (via RSS or email) to get it delivered to you.
• Weekly “30 seconds of VMware”.  We use these internally to note important bugs, blog posts, launches, whitepapers.  Previously only available inside EMC, we’re now sharing this with the customer and partner community openly.   Subscribe (via RSS or email) to get it delivered to you.
• Calendar of events.  There’s literally so much going on every week (webcasts, VMware Knowledge Series, VMware View tour, EMC Forums, customer councils), that’ it’s difficult to keep up to date.   Here’s a simple place where you can see what’s going on that you can leverage. 

• Demo tools.   While I try my best to be a good source here at VirtualGeek, it’s just me talking, and you commenting.   at the “Everything VMware at EMC” community, you can make requests for demos/tools, and anyone can post something cool!
• Breakdowns of topic around the “pillars” of EMC’s business – Storage, Security (RSA), Backup/Recovery (BRS), Management (Ionix)
• Contests and prizes… And here’s the first.   It’s right there on the column on the right of the “Everything VMware at EMC” community… Share your story of how you use VMware, EMC, and Cisco technology together and win (not “enter to win” in the sense that you statisically may or may not based on chance, but until I run out – every story wins) an Iomega ix2 or ix4.   We have loads of customer success stories, but each “lights up my life” a bit – after all, it’s the customers that make the world go round….

The most important criteria as the team put this up there were:

1. wide open.
2. easily searchable.
3. can post in a heartbeat.

So – it’s now up to us.  Will it suck, and die, or blossom into a great tool for us all?   Reminder to be good – if there the content is public, link to it.   If you cut and paste something, attribute it to the author.   EMCers, you cannot use this to bypass the rules that govern confidential content (hint, if noted as confidential, it can’t be posted).

BTW – if you like this idea, but aren’t into this particular community (though if you’re reading my blog, you’ve already kinda “self-selected” as a VMware nut :-), there are other similar ones:

http://www.emc.com/community/

I personally dig the Celerra one (has APIs, SDKs, and much, much more)….

Fixing eagerzeroedthick use - In VI3.5, vSphere and using Zero Reclaim.

As discussed here, in VI3.x, any VMs deployed on VMFS via “clone” or “deploy from template” by default use the “eagerzeroedthick” format.  This was the case all the way up to update 5, which fixed this behavior.   vSphere 4 “clone” and “deploy from template” operations have always used the zeroedthick format (exception is a VM configured for Fault Tolerance, or where the administrator forces the eagerzeroedthick option – for a MSCS/WSFC device for example).

This is important for the reasons covered in that post (check it out) – but the key is that it means you use more storage (a lot more) than needed.

I’ll add another reason – it literally doesn’t just consume more storage, but it makes takes that do this “zeroing” (clone, template) take a lot longer than they need to.

Long and short, even if you do thin provisioning at the array level, eagerzeroedthick VMDKs “cancel it out” – because they “pre-zero” every portion of the VMDK.    Note that production storage dedupe or compression techniques can solve this (since all the zeros can be eliminated), but thin-provisioning at the array level on it’s own cannot.

So, how can you fix it?

• One option to resolve this is to upgrade to vSphere and use the new Storage VMotion.  If you storage vmotion an eagerzeroedthick VM and select “thin” or “thick” you end up with a “thin” or “zeroedthick” VMDK in the end, both of which are more storage efficient.   Also, starting in vSphere, all clone, template operations always default to “zeroed thick”
• Some arrays can do a “zero reclaim”.   This takes virtually provisioned storage devices and “frees up” these “all zero” extents.  This is one of the nice (and free!) little tidbit introduced with the latest Symmetrix Enginuity code (applies to DMX and V-Max) is a “zero reclaim”.   BTW – if you want more on this, you can read up at Barry’s site here.  This looks for any extent in a virtual (thin) pool that is entirely zeros, and frees up that extent for other use in the virtual (thin) pool.   Note that this feature is OS-agnostic.   It will free up “all zero” extents regardless of the host type.

Here’s a quick demonstration of the Space Reclamation (aka Zero Reclaim) as applied in the VMware use case, and also showing how to configure it in the Symmetrix Management Console (SMC).

You can download the high-resolution WMW here.

• If you’re on VI3.5, you can also fix it – independent of array type.    BTW – kudos to the EMC and VMware engineering teams here.   It would have been too easy to just say “upgrade to vSphere 4” or “make sure your array has a zero reclaim”.   The EMC team pushed hard based on customer demand, and the VMware team delivered.    You need to do some specific things to make it work: a) be running ESX 3.5 U5 (Build 207095); b) be running Virtual Center 2.5 U6 (Build 214388); c) configure the ESX host as noted in this VMware KB article: http://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=1017666.   Here are the release notes for U6 with a little more info: http://www.vmware.com/support/vi3/doc/vi3_vc25u6_rel_notes.html#resolvedissues

We’ve seen many customers do either of these approaches on their on their VMware environment and reclaim a LOT of space (40-50%) – and that’s a GOOD THING.

VMware Partner Exchange 2010 from where I sat…

Phew – after 5 days in Vegas, you get pretty cooked.   At home now finally and love seeing my family.     Well – a quick little summary of the week, and what we announced, showed, and discussed.

Was a GREAT VMware Partner Exchange (PEX) – thank you VMware!

So – what did we see and do?

• VMware’s continued growth is one of the largest drivers for partner growth.   PEX attendence was up 77% over last year.   
• VMware is super-focused on the partner community.  This came through loud and clear in Carl Eschenbach’s keynote.
• We saw this too in the EMC bootcamp.   We held a bootcamp on the Monday of the event for the EMC partners present.   There were almost 200 people there all day long.   Thank you EMC partners!  Would love your feedback on the event.
• The VCE roundtable was packed – with 202 partners.   Got great questions, and great feedback.     Cisco’s posting the video shortly.
• The VCE reception was also fantastic, thanks everyone for attending.
• My team: there were a bunch of new members of the vSpecialist squad – was great to hang out with you!
• The EMC booth: was cool that we pulled together the Vblock 0 prototype to show at the show, and got positive feedback on the demos.    Good team on the schwag too – those little wind-up flashlights will help deal when the inevitable apocalypse arrives :-)  Oh, my kids will like them too :-)
• TAP: Got solid feedback on the vSphere roadmap and Storage Ecosystem roadmap sessions from some of the team newbies (of course, the vets all were in the loop on this already)
• VCDX defense for some of the folks on the team.  Scott, good luck, I’m confident in you!
• Steve Herrod did some big unveiling.   Project “Redwood” (end user self-service portal targeted for Private and Public cloud uses) was public outed for the first time, as well as the next version of VMware View (loads of stuff in here, more to come soon).  He also talked about the scaling and feature goals of the next generation of the vSphere generation.    Even with all the legalese and caveats at the front of the session, it’s very exciting stuff.    My team and I are lucky to have insider front-row seats to everything that’s been going on – so none of this was a surprise to us, but it’s great that it’s getting out there.
• Fun stuff:
  • Parties: the tailgate party Cisco and EMC sponsored, and was loads of fun!   Great exciting Superbowl game.   I have to say that personally, I think The Who can still rock. 
  • Playing craps with friends – I hit 4 straight yos when it counted, and the money was raining in.    Also on the last night with some new friends where I was sucking it, but man, they were rolling like nobody’s businesss….
  • The big party was fantastic – at the House of Blues, with a great 80’s band – the English Beat
  • The VMware EBC on wheels – just awesome…. Pictures on that one below.
  • The VMware/EMC/NetApp alliance team dinner :-)   Pictures and the story on that one below!

In the EMC Bootcamp we spent the day on helping partners get more out of their business, and making sure they had all the latest tools we provide to help them help their customers.   I did the keynote which was “VCE: What’s going on behind the scenes and how can you make the most of it”.   This frankly discussed where we are (including gaps) since the VCE launch, as well as providing a technical preview into the integrated 3-company roadmap for the next year.

I also showed the new Celerra NFS datastore VMware capabilities, previewed the next version of the EMC storage Viewer, and the next version of Unified Infrastructure Manager (UIM v2).   I also gave some big hints on the stuff that’s coming at EMC World – each of which will be huge….

If you’re interested in more detail, including demonstrations and screenshots of these new and “arriving so shortly it’s essentially now” functions that we talked about, read on….

EMC Solutions Enabler – available as a vApp

EMC solutions enabler 7.1 is out and directly available as a vApp.

If you login to Powerlink (EMC partners, customers, employees) you can find it here:

Home > Support > Software Downloads and Licensing > Downloads S > Solutions Enabler

Here’s a screenshot of the download link. 

Personally, I’ve found the vApp package makes deploying solutions enabler (set of tools, public APIs used by many storage-related management tools) much easier.   Remember that if you are using a Symmetrix with Solutions Enabler that you can easily configure gatekeeper LUNs (small LUNs used for inband communication to the Symmetrix) using physical mode RDMs.

Fun with Vendor FUD – Episode #1

UPDATE Feb 12th, 2010: 10:52pm ET:

I really do want to reiterate, I posted this in a light spirit, some of the claims struck me as funny ("The only SAN with vSphere vStorage Fault Tolerance support included at no extra charge"), and some so demonstrably incorrect ("The only SAN platform which supports SRM for both automated failover and FAILBACK") that I laughed, and my team laughed.

So, why not spread the laughter?  Don’t read into any more than a fun thing to make us laugh.   And yes, if you get very literal, it’s not technically FUD if it’s not directly negative on another vendor, but hey, then the alliteration in the title of the post doesn’t work :-)

UPDATE Feb 17th, 2010: 1:16am ET

Thanks for all who responded!   I’ve closed the survey, and gathered 209 responses from around the world.  The answer to the question “who said this?” is below.

There’s so much fud that gets slung around these days (not implying EMC is immune!!!) that rather than get frustrated by it, my team tries to have a bit of fun with it.   I think the game is fun enough to share out there :-)

Ok, this first variant we call “name that vendor”.   A question comes in from a customer, which is clearly a “list of ‘unique's’” that was sent by a vendor.  By “unique’s” I mean a list of pithy points where a vendor makes a grandiose claim of uniqueness on a given feature in the hopes the customer makes that one thing on that list a buying requirement.

What we like to do is strip out the customer name, and the vendor source, and play “name that vendor”.  It’s always fun because it’s usually from sales, and they so mangle what the product actually does do that’s cool, or make just flat out incorrect or silly claims.    Since VMware is one of the most popular customer use cases of infrastructure, we get to see some fun ones…

Here’s the first Virtual Geek episode of “Fun with Vendor FUD – Name that Vendor”! (I’ll answer the source after a week of polling)

• The only SAN with vSphere vStorage Fault Tolerance support included at no extra charge
• Full Application aware API integration with VCB at no extra charge
• Clustered SAN architecture to match your ESX server cluster
• Fully integrated MS VSS apps with API hooks
• Full Synchronous replication software included at no extra charge
• The only SAN platform which supports SRM for both automated failover and FAILBACK

Survey has been closed – don’t worry – there will be more episodes of “Fun with vendor FUD” – survey result (209 votes) – results below (including geographic dispersion of votes)

There’s all sorts of vendor craziness out there.  

My advice?   Ignore claims of “uniqueness” (heck, even from EMC in general, or me specifically).   Look at the product on it’s own merits.   What problems could this help you solve today?  Does it have value that is greater than it’s cost.   Does it align with where you see IT going?  

“uniqueness” in any industry is hyper-transient.   “Different”, well – every product has architectural elements that mean that it can do things differently – and that’s legit, and customers should consider them.   My other advice?  Have some Fun with Vendor FUD :-)

If you think this is funny like I do, I’ll keep posting more episodes…

…So – who said it?  

First of all – don’t take this as an indictment of any vendor – all of us do these shenanigans periodically.   And also, remember that this truly is in the spirit of fun.  I will try to be even handed, including “Fun with Vendor FUD” episodes that poke fun at EMC.

… so – in this case, the source was from an HP/Lefthand.   The giveaways (to me) were:

• “The only SAN with vSphere vStorage Fault Tolerance support included at no extra charge”.  I think this is a rep confused about the fact that you can run the Lefthand VSA, and apply FT against it (though of course, they software RAID and have a distributed volume manager, so you don’t use it that way – you run multiple VSA nodes?).   vSphere Fault Tolerance works on all storage platforms on the VMware HCL.
• “Clustered SAN architecture to match your ESX server cluster”.  This one is a funny one to me too – while there’s an argument to be made about "scale out of x86 servers with DAS with software turned into a scaled out storage array” (note that interestingly a V-Max is literally a set of scale-out x86 server with storage attached – just with the idea of linking all the servers into a common memory space so cache is global/any IO served by any port/brain via the Virtual Matrix) is a good design, there’s nothing implicit that makes that better for vSphere use cases, more or less than any other use case.  Remember I came from a company (Allocity) that did a similar idea (though at BEST we were a much, much less mature implementation than Lefthand), so I get the value, and the downsides of the approach.  

This one is flat out wrong.

• “The only SAN platform which supports SRM for both automated failover and FAILBACK”.   This is wrong, EMC offers this on 3 of our 4 SRAs, and I believe that EqualLogic and Compellent offer something similar.

Look, I’m not trying to stir the pot here.    Honestly, every vendor does this, and every sales rep is always looking for “silver bullets” to put out there.  I’ll continue the series so long as it’s in good fun, and people don’t get too bent.  I’ll try, as always to be as even handed as possible.     Was interesting to see the votes – NetApp and EMC got lots of votes (though the statements too me at least pointed to EqualLogic or Lefthand) – suggests perhaps we both have a reputation for a lot of FUD?

Some important Celerra and Symmetrix SRM updates

There are two important updates to the EMC VMware Site Recovery Manager (SRM) storage replication adapters (SRAs) for our joint VMware/EMC customers I wanted to make sure everyone knows about.

First, the Symmetrix SRDF update.

A maintenance release SRDF SRA v2.1.1 was certified yesterday by VMware and is available now for download at: http://www.vmware.com/download/srm/. The downloaded zip file includes also updated the Release Notes.

The major fix for v2.1.1 is:

1. Proper handling of configurations using 64-bit builds of Windows 2008

Also note that SRDF replication state is shown (even configurable) in the latest EMC Storage Viewer vCenter plugin (v2.1)

Next, the Celerra Replicator update – v4.0.17.

A maintenance release v4.0.17 was certified several weeks ago by VMware and is available for download at: http://www.vmware.com/download/srm/. The downloaded zip file includes also updated the Release Notes.

The major fixes for 4.0.17 were:

1. Proper handling hyphens in interconnect and replication session names
2. Filtering out  internal datamover IP addresses that are invalid for NFS mounts as available storage ports
3. Adding local replication to our session filter so that parsing isn’t broken

Problems found since 4.0.17.

1. (NFS and iSCSI) Long interconnect names (longer than 17 characters) break parsing, making it impossible to match destination LUNs to their primary LUNs
2. (NFS) Unable to specify exactly which datamover IP addresses to use for NFS mounts.   It is valid to use a particular IP address on the datamover for replication and not for exposure to ESX, however, if the SRA exposes this IP as an available storage port then SRM will attempt unsuccessfully to mount it during test and actual failover.  At that point, SRM will report an error to the user though the test or actual failover was successful if SRM used the correct IP address.

EMC and VMware are currently testing 4.0.19 for certification in the next few weeks.

Remember - please make sure that you are using the latest officially supported SRAs – which should always be downloaded from VMware’s site: http://www.vmware.com/download/srm/   I’m emphatic on this one on purpose.  There’s a ton of certification that goes along with these updates, and while “pre-release” versions always circulate around, they should never be used in a production environment.