Virtual Geek

2 extreme proof points from the NAS leader…

So – today, there were two releases that represent two totally opposite ends of the extreme when it comes to NAS (and suggests, perhaps, the way things will go over time).   Interesting to see how it represents where EMC is today (and hints at where we’re going) re the extreme entry-point, and extreme scaling point.

First – Dr. Martens (yup, the folks with the cool shoes) picks Iomega for their relatively modest (but still critical to them!) NAS and block storage for both primary and for helping their backup/recovery processes.

Dr. Martens has taken advantage of its Iomega StorCenter ix4-200d units to perform faster, flawless backups.

Dr. Martens also plans to leverage the Iomega StorCenter ix4-200d's robust, reliable performance and iSCSI connectivity to expand its reach within the organization. Additionally, the fact that the StorCenter ix4-200d is the first NAS server in its class to be VMware certified gives the company an easy and highly cost-effective replacement for its aging IBM SAN. "The Iomega StorCenter ix4-200d is exactly what I was looking for," concludes Guanco. "This highly functional NAS platform has exceeded my expectations in terms of supporting disk-based backup, recovery and disaster recovery. I'm looking forward to additional uses in the future."

Second – EMC Isilon doesn’t just break, but SHATTERS the world records for NFS and CIFS performance and scaling.

You can see the SPEC SFS results here, but here’s a summary table of some of the key results:

Vendor	Peak CIFS IOPS	CIFS ORT	Peak NFS IOPS	NFS ORT	Number of File Systems
EMC Isilon	1,612,778	2.34	1,112,705	2.54	1
EMC VNX	661,951	0.81	497,623	0.96	8
Huawei Symantec	712,664	1.81	636,036	0.99	8
IBM	--		403,326	3.23	1
NetApp	64,292	1.50	190,675	1.17	2

You can see that EMC is doing great here in two distinct categories.   VNX holds the record in the “swiss army knife” grouping (storage platforms that do it all) – which consists of few others, but certainly our respected colleagues at NetApp (FAS hardware running the ONTAP 7-mode software stack).

There’s the emerging category of large scale-out filesystem storage, where Isilon competes with some (IBM SONAS, and NetApp FAS hardware running the ONTAP cluster-mode software stack or NetApp Engenio hardware running things like Gluster), and competes very strongly.  

A little sidebar… NetApp – you are our primary NAS competitor, and a strong competitor for EMC in the “swiss army knife” category (VNX vs. 7-mode software stack on FAS hardware).  But clearly these massive scale-out global namespace models is one of the ways the filesystem world is evolving (in parallel to continued evolutions in the “swiss army knife” category).  Where are you?

Ok, back to the point… The key, however, about the Isilon result isn’t just that it’s huge (it is), or that it’s a single, global filesystem that can be PB in scale (it is), or that it’s 140 nodes in the cluster (which it is), or that it uses swank Isilon S200 hardware (Westmere, SSDs, SAS, 10GbE out the ying-yang).    

What’s a amazing to me is that:

1. the environment scaled linearly (and results were posted at 14, 28, 56 and 140 nodes scaling points)
2. customers can start small (and low cost) and easy.
3. and it stays easy as the cluster grows.   Filesystems spread out automatically across nodes as you add them.   Metadata gets distributed across nodes – including using SSDs preferentially where they help.   If you’ve never added a Isilon node – a trained monkey could do it.  Heck, forget the training – give it an IP, and go.  

Here’s a video that shows the cluster growing during the build for the SPEC SFS exercise.

Pretty darn cool if you ask me :-)

June EMC/VMware Webcasts–solid…

In June – we’re covering VMware-centric backup, VMware/EMC sharing best practices for Oracle on vSphere performance and provisioning, a Unified Infrastructure Manager update (cool), Using Recoverpoint in Vblock deployments and then Chad’s Choice (which I do once every quarter).

There are a couple doozies already down (backup and Oracle) – though they are ALWAYS available for watching.

For this quarter, “Chad’s Choice” will be: “3 Pieces of Bad News and A Deep Dive into All Paths Dead”.  

May sound academic – it’s not.   It is CERTAINLY going to be marketing-free.  I’m going to try something a little new – spending time on 3 recent VMware/EMC issues (VMAX, VNX, PP/VE), and discussing what they are, how they happen, and what we (VMware/EMC) do about them (input welcome).   In there, there will be detailed discussions over the bugs themselves, how to use VMware/EMC communications (as a customer or a partner) to your advantage.    

The second topic is important for any customer using any vendor’s block storage with VMware.  The All Paths Dead (APD) behavior has changed a lot, and will continue (hint hint) to change in the future.   Understanding this will help you as a customer/partner make your vSphere environment more resilient.

Click on the below to register (or watch sessions that have already happened):

New VNX/vSphere Techbook published…

This is the authoritative EMC guide to VNX and vSphere.   Check it out here:

This one has updated guidance on sizing (hint: make big datastores see page 43), updated guidance on SIOC (hint: always turn it on, see page 46), updated guidance on parity (hint: use parity RAID for your “generic datastores”, and focus your efforts on specific design for the heavy-hitting mission critical apps).

Nice to see it out formally, our guidance continue to get simpler.  The general principles align with Chapter 6 (Storage)of the “Mastering vSphere 4” that Scott Lowe wrote (and I guest wrote the storage chapter).   Simple, big datastores + just get going is the right way – and knowing what to measure and react (since you can adapt non-disruptively) is the right way.  Only spend time on focused planning for VDI (IO intensive) and mission critical apps (IO intensive and SLAs).

HINT: the next rev for vSphere.ahem is well underway.

IMPORTANT: VMAX epack and vSphere hotfix

Ok – I made a post here about a couple of issues.  The first wasn’t really an EMC issue per se, but since the elab support matrix is followed to the letter, and there was a known timing issue with an Emulex firmware release, I wanted to point it out.

The second (in the span of the few days since the post) has popped up in enough places (still only a few, but it doesn’t matter – rare or not, customer pain is bad) that everyone at VMware and EMC went into furious high gear on this over the last few days.

So – an ETA (these are EMC Technical Advisories) went out on June 7th.  This goes to all EMC customers, but I wanted to draw EXTRA attention to it.

The core issue is related to VAAI and VMAX arrays running Enginuity 5875.135.91 or 5875.139.93.

Remember, since VAAI is on by default, if you have upgraded to vSphere 4.1, and your VMAX to Enginuity 5875 (those specific revs) – this applies to YOU.

The detail of the issue, and workaround are posted in EMC primus (http://knowledgebase.emc.com) article emc263675.

While I encourage customers/partners to read the ETA and the primus article here’s the key of the fix:

• Do NOT use the VAAI Copy offload function until an available Enginuity Pack is loaded. An alternative is to disable the advanced parameter DataMover. HardwareAcceleratedMove by setting the value to 0 (the default is 1).

  Note: You can disable VAAI across your vSphere clusters using this powershell script (thank you Nick).   This requires rebooting the ESX host, but of course using vSphere’s maintenance mode, you can do this non-disruptively. UPDATED 6/16 10:50am ET – a bit of a space-out moment.  Of course, changing the ESX datamover from software to hardware (changing the advanced property) doesn’t require a reboot.

• An Enginuity Pack at the current Target code, 5875.139.93, is available to prevent this issue. It should be loaded to replace 5875_135/0084 / 5875_091/5011. The new Enginuity Pack adds fixes 56516, 58269, 58328, 58546, and 58566 at 5875.139.93, R/B 5875_139/0013 and is available from the SSR GA Enginuity Pack download page. Environments at Enginuity 5875.135.91 and 5875.139.93 using VAAI should load the Enginuity Pack and the VMware patch at the earliest opportunity.
• In addition to loading this Enginuity Pack, customers must also contact VMware Support directly and request a Hot Patch. This hot fix is required to fix an issue encountered when using VAAI for Storage vMotion and Cloning operations where performance degradation can result due to the the operation(s) taking longer to complete than when using Software (Host) Data Copy.  VMware is currently working to release this fix in a GA version of ESX 4.1 targeted for release mid Q3 2011. However, if you encounter this issue, report the problem to VMware Technical Support to receive the Hot Patch.  All customers with a VMware Support contract can file a support request using the instructions on the support site: http://www.vmware.com/support/contacts/?x=42&y=13.

For what it’s worth – my apologies to customers, we should have caught this pre VAAI GA. 

There’s a lot of post-mortem that has gone on, and continues to go on, and we have changed our going forward testing and qualification model around things like this to improve. 

While it doesn’t make up for a miss, we try to hold ourselves to a high standard, and that means being blunt, owning up to the error, being honest and self-critical, and then fixing the core process.

Again – thank you to our customers for being joint VMware and EMC customers!

Chad’s World Live (from EMC World ‘11) – behind the scenes

This was a ridiculous amount of fun…  and a first – Chad’s World was live at EMC World 2011.   This was a ridiculously stupid attempt – Wade and I are not good actors, and normally need many takes to do anything right :-)   Our hope was that while it could be an epic #FAIL, at least it would be a fun & irreverent way to wrap up EMC World, and open up for the Fray.

For what it’s worth – as much as this is a gas, I want people to know that while I’m an exec at EMC, more importantly I’m a dude who puts his pants on one leg at a time – just a normal guy who happens to be a passionate nerd.  Our marketing geniuses (and they are geniuses) thought this would be a funny way to entertain and educate on stuff EMC is doing, and leverage my Canadian-ness and the fact that yup, I do have a large home lab in my basement :-)   I’m also completely willing to make an ass out of myself, which seems to be a pre-requisite for something like this!

A big thank you to the customers – first for being EMC, VMware, Cisco customers, and also for being such good sports to join us and share their killer stories!

• Jonathan Visser @ Morris Hosital
• Chris Kudlick @ Morris James, LLP
• Mike Hurst & Rael Mussell @ Credit Acceptance Corp
• Tom Pearson @ Loves Country Stores
• Kyle Berger @ Alvarado Independent School District
• Haim Inger @ CLAL Insurance
• Brendan McGowan @ Safe Systems

Their stories of how they are using our technologies together are just amazing.   Frankly – I think they make the show worth watching.

Also a big thank you to VMware, Cisco, Intel, Wyse and Molson’s for all supporting the event!

Click on the picture below to get to the streaming video (all Chad’s World episodes are at www.emc.com/chadsworld), and read on past the break if you want the “insider scoop” of what was going on behind the scenes.

Also – here’s a bunch of pictures of the event (including some behind the scenes)

For people wanting the “behind the scenes” (ergo what was REALLY going on as we’re on stage), read on…