EMC World 2016: Unity… the story BEHIND the story.

[UPDATE: 5/7/2016 @ 10:23am ET – a couple corrections, inline compression is targeted at 2H’16, not Q2.   Also see comment section for deduplication dialog]

This is, in my personal opinion, one of the coolest stories at EMC World this year – and let me tell you why. What do you think is harder: inventing something new with a clean sheet of paper, or changing the entire car as it’s barreling around a racetrack?

There’s a joke in the IT industry that “The reason God was able to create the world in only six days was because he didn't have to deal with an installed base.”

I think what Unity represents is an incredible engineering achievement – and an incredible story – a wholesale architectural change, while moving, with a massive installed base.

This post will cover what’s new in Unity, but the thing that the most fascinating is the story behind the story. I’ve been lucky to see it from the inside, and today I’ll share it (from my perspective) with all of you.

EMC VNX has an installed base that is well, well north of 100,000 deployed arrays. It has been the most successful EMC platform to date in terms of the number of people and customers, and VNX brings more net new customers to EMC than anything else we do – even as it became increasingly aged.

The VNX was released on Jan 18^th, 2011. That’s more than 5 years ago. For perspective, that’s when the first MacBook Pro was released, and about a year before Windows Server 2012.

Since then – it’s been a huge success.

Now – even before the VNX was born, we were already grappling with how we would transition the architecture and move completely past it’s CLARiiON and Celerra roots, and their deep architectural underpinnings – we needed to move away from the old, but how?

There are other examples of “change the engine while the car is running” in the storage ecosystem playing out, and it is NOT EASY.  I’m not saying we’re perfect, but this is an interesting example that has been occurring in parallel, somewhat “under people’s noses”.

Unity is that – and it’s a whole new platform – we have swapped every part out while not missing a beat in VNX/VNXe. 

Unity will redefine the mid-range unified market.  Its faster, denser, less expensive, and simpler than all of it’s competition (and we’re willing to put our money where our mouth is with a money-back guarantee). 

VNX was the king in this space.   The King (VNX) is dead (though not EOL!), long live the King (Unity)!

For more gory technical detail, and the fascinating story behind the scenes – read on!

As I’ve noted before – it takes a lot of time to harden a storage stack from inception. A rough rule of thumb is 5 years for a transactional stack, 7 years for a NAS/Object stack. Scale-Out/distributed variations are materially harder. This isn’t a matter of “smarter engineers” – it’s rather that persistence stacks are HARD, and I hate to say it, but there’s an incompressibility of the schedule because in part that time is because they get hardened at customers, in deployments, over time.

When you’re a big company (vs. a startup) this is a little terrifying, because you don’t get the “protection” of growing the installed base slowly. For example, as Isilon and XtremIO have ramped like crazy – they both have gone through bumpy periods before they stabilized.

How do we do it for the most widely deployed enterprise storage stack (VNX) on the planet – while minimizing customer and business risk????

As a business, we embarked on an long journey with important milestones that culminate today.  I want take a moment to give a shout out to Rich Napolitano under whose watch this started, and the hundreds of engineers who have invested blood, sweat and tears over the years – what a journey it’s been.

This has been a journey to redefine, recreate the world best mid-range external scale-up unified storage platform, and do it without risking customers, partners, and revenue. You can literally see powerpoints of the internal strategy in the bottom of this post from 2011.

The journey can be summarized this way (and it’s fascinating to any people interested in this part of the industry – you have to play the long game):

• Step 1: build a new core platform at the kernel level that would allow us to modularize the code, and support containerized packages, as well as clean kernel-mode insertion of things that need to be very close to the hardware. This project was codenamed “C4”.
• Step 2: find a way to test that out (and start hardening) with customers. Thus was born the VNXe in 2011. You can see this milestone in the journey here. The VNXe has turned out to be a huge success – bringing more new customers to EMC than anything we’ve built before.   There are still things smaller than a VNXe, and Unity, but that’s another story for another day.
• Step 3: At the same time that you do Step 2, completely do a ground-up re-write of the block stack. Start changing the design center in anticipation of pervasive flash, and multi-core CPUs – this was jettisoning the old FLARE code base that VNX inherited from CLARiiON, and replacing it with MCx. Without these changes, legacy very “simple” mapping models would inhibit new data services like compression/dedupe, and would leave the platform in the 1990’s in terms of snapshot behaviors and LUN ownership/failover behavior. This was born the VNX2 in 2013 here. Like VNXe, the VNX2 has been a wild success – billions of dollars of revenue, hundreds of thousands of customers.
• Step 4: completely re-write the NAS stack, which was mature – in both good and bad ways. Good – very robust, and with all the stuff enterprises expect. Bad – 32-bit filesystem (seriously!) that meant small filesystems and filesystem limits, a legacy mechanism for filesystem meta-data mapping that meant that NAS/block functions were very disparate, and failover behavior was too slow. What was needed was a modern 64-bit NAS re-write that mapped into the new block mapping code.

Each of these steps were carefully coordinated, and consciously broken into “digestible parts” so that each part would get hardened, tested – and we would ultimately change the whole car while barreling down the highway.

That journey ends today – and a new journey with the new platform, Unity, begins.

Unity will cover the entire market segment that VNX and VNXe currently serve. Well – not entirely. We’ve realized that at the top-part of the VNX bands (VNX8000), scale-out models will rule the roost – and there XtremIO and VMAX All Flash rule the market (expect them to have configurations that move down to cover the top part of that market).

Unity is squarely focused at the external storage use cases that start small (into the <$10K bands, and with all-flash configurations that start at $18K) – and go up to 500 SSDs/HDDs.

How small is “start small”? Well – with Unity – we are covering the VNX market as a whole with the system architecture that looks like a VNXe. 2U, and everything is in there.

 

How powerful? In practice, about 3x more performance than the previous generation at every scale point, and not just more powerful, but more dense, more simplicity, less power consumed – not a small leap, but a huge leap. Wow.

And the core codebase isn’t just faster, it’s also better.   New mapping, snapshot underpinnings, and a killer advancement in the QoS behaviors where there much finer granularity, and the ability to provide distinct QoS handling for primary devices versus replicas.

Now – if you look at the picture - you’ll notice there are NO “datamovers” or “file blades”. The convergence is complete.

Speaking of the filesystem – I’ll say this strongly: this ain’t your daddy’s VNX.

Unity is a scale-up system – and is EMC’s scale-up transactional filesystem (scale-out non-transactional is OneFS) we came to the conclusion that the requirements for each were different enough that to try to do both could (would?) result in “flying-boat” syndrome – something that is not particularly good at either goal.

The next-gen filesystem goodies (including the neat VMDK on NFS implementation) will arrive to VMAX customers as well via the embedded NAS (the power of containerized storage stacks :-)

And there’s more to the new filesystem than just new “features”. Filesystem fundamental limits are all up – way up!

Furthermore the – while VNX filesystem resilience has always been good, the hardening that has been done is huge. Fast filesystem start and restart, all metadata is checksummed.

Also – with containerized NAS stacks – we can support full multi-tenant use cases, including segregated IP spaces immedaitely, and even overlapping IP address spaces – historically a gap in VNX relative to some of it’s competition.

Again, a huge leap (you’re seeing the pattern :-)

How exactly do we DO all of this?

Answer: it’s all about software - the Unity Operating Environment. All the years of work on “C4” containerization and abstraction, modularized MCx and unified next-generation filesystem (which doesn’t “layer on” LUNs, but directly intersects to the block mapping layer – so the dataservices like snapshots are completely unified across block/NAS use cases)… Well, it’s come to fruition.

All this runs on a thin, light, hardened version of Suse Linux. No FLARE in sight :-)

And if that all makes our customers happy – this will make them dance in the streets: NO MORE JAVA. The entire UI is HTML5 – simple, works nearly everywhere, and is FAST. In fact, we wanted to make sure that we we’re drinking our own Koolaid, so had MadPow put Unity through its paces for full lifecycle tasks – and compare it with others who are well regarded for their usability/simplicity.

We didn’t just stop at the simplest UI in the business, we also built in a ton of new capabilities to make support better.

 

We’ve improved on the already great embedded support on VNXe, adding CloudIQ – which is a cloud-based lightweight reporting and management tool that has to be seen to be believed (Modern Datacenter Architectural principles apply here – flash optimized, and cloud enabled).    EMC attendees – make sure to hit Susan Sharpe’s session on this topic!

BTW - if you need more reporting and analytics than “comes in the cloud” via CloudIQ, EMC Storage Analytics (using vRealize Operations) will give you everything you need.

Unity is a family – available in “F” variations that are optimized for the all-flash datacenter, and more traditional hybrids. The space for hybrids is shrinking fast – and we believe the all-flash configurations are the sweet spot and have hit the inversion point for most customers. With 3.2TB SSDs available now, and 15TB SSDs around the corner – this trend is not going to stop.

 

Also worth noting – the Unity Virtual Appliance is also available – and you can download and start using it TODAY here.

With the Unity VSA- there is a version that follows the new EMC model of “free and frictionless” (go to town, just don’t call for support – it’s community support only), and there’s a commercial version for customers who want support for things like edge deployments that can nicely replicate back to the core datacenter.

And of course (happy to say this as the leader of the Converged Platform Division, VCE) – there is a new Vblock platform sporting all this new bad boy – the Vblock and VxBlock 350.   For customers who want to continue with VNX-based Vblocks, the older Vblock and VxBlock 340 systems remain orderable (sometimes it takes time for customers to pick up new standards).

As you can imagine – Unity means we can make the Vblock/VxBlock 350 smaller, and more cost effect (and improve performance).   The performance optimizations we’ve made in the Vblock/VxBlock system design to adapt for all-flash configuration (which strains fabrics) like the Vblock 540 (XtremeIO) and the Vblock 740 (VMAX All-Flash) fit well in the new Vblock 350.

Unity also expands on the deep ecosystem integration VNX2 and VNXe enjoyed – in particular the VMware and Microsoft ecosystems.

Unity directly connects to vCenter for VM-level awareness and mapping. Unity has updates on the VAAI implementations (make sure you look at updated best practices).

Of course – Unity also enjoys unified VVOL support with Protocol Endpoint (PE) support on block and file:

Beyond what’s in Unity itself, Unity will inherit all the Integrations that VNX enjoys with Virtual Storage Integrator (VSI) for VMware, EMC Storage Integrator (ESI) for Microsoft, AppSync (for app-integrated replication), and EMC Storage Analytics – with these dates as our targets:  AppSync (7/11); VSI (5/31); ESI (now!); and ESA (now!)

We also have listened to customers and learnt from the marketplace that we needed to simplify licensing and the whole economic package. Just like XtremIO and the VMAX All Flash – Unity uses a simple appliance economic model. In other words, everything is included. No “per GB” licensing of software.

Beyond “all included” – how about “no maintenance and support jack-up”?    This dramatically simpler model is de rigeur in the all-flash array market, and since we’re leaning hard into all-flash across the portfolio, and that model is used in XtremIO and VMAX All Flash – it will apply to the all-flash Unity as well:

Holy smokes. This ain’t your daddy’s EMC :-)

That’s a whole lot of “new” here.   In the diagram below – every green arrow represents something fundamentally new in Unity.

And the Unity team isn’t stopping here. 

In 2H 2016 there will be an NDU software update that will add inline compression – just like we are with VMAX All Flash – which will have a effective capacity multiplier of 1.5x to 2x.   A nice little “thank you” – and highlights all the under-the covers core code stack changes.  Our earlier efforts to do compression were a less-than ideal post-process, and with material performance impacts.

I would argue that between VxRail and Unity, we have perhaps the two simplest products for customers to use, for our partners to embrace… And imagine the power of both VxRail and Unity in the hands of the combined Dell/EMC entity (where integration of the powerful Dell supply chain and server portfolio is a huge force-multipler).

Sidebar: Speaking of VxRail – there’s a lot of internal (and external) debate about the storage market transitions – and what SDS and Hyper-converged Infrastructure Appliances (like VxRail) mean to the “mid-range” unified storage market.

What is a fact (data, not opinion!) is that the external mid-range external storage market currently dwarfs (by a factor of more than 10x!) than the total revenue of all the HCIA players and SDS players put together.

There are also plenty of scenarios where the economics ($/GB, Watts/RU, GB/RU, $/IOps, $/GB) strongly favor external storage models over HCIA and SDS models. It’s a fact. Do the math :-)

I’m sure that over time, we’ll see HCIA and SDS models continue to grow and expand – and cannibalize the external 1storage market, including the mid-range. But – that will play out over years, and the customer needs for simple, scalable, performant, reliable external storage are everywhere.

With all this awesome new Unity goodness, our hundreds of thousands of happy VNX customers may be wondering what it means for them.   VNX isn’t going away – and this doesn’t mark it’s EOL to say nothing of it’s EOSL (if you want to know the EOL/EOSL for any EMC platform it’s right on EMC Online Support under your products).

We know that our larger customers take time to qualify/certify new products, but starting today – if was in the market for a mid-range, unified platform, something with great transactional NAS – I would evaluate Unity, period.  Today marks an incredible new beginning.

To all of the players (traditional and the startups) in this section of the market – there’s a new player on the scene, and Unity is loaded for bear, and competition is fun and good for the marketplace!

More importantly – customers – what do YOU think?

Comments

[Moez]

What happened to deduplication? Will it be added later?

[Mark Burgess]

Hi Chad,

It certainly looks impressive.

I had a couple of questions around compression and de-dupe:

1. You say Inline Compression will be available in Q2 (i.e. by the end of June), did you mean Q3?

2. What is the story around block de-duplication (either carrying forward the post-process VNX2 technology or something new (i.e. inline))?

I have heard that the focus will only be on compression, which does not seem ideal, especially for Unity all-flash as I would have thought it would benefit from de-dupe (most competitor AFAs have de-dupe).

Many thanks
Mark

[abj]

Will Unity offer block dedupe or will it still be File only?

[PT]

What does Lifetime* mean?

[Frank]

How about online data in place storage processor upgrades? What is the Unity all flash option using under the covers to help protect against SSD wear if your using traditional RAID?

[Chad Sakac]

[UPDATE: 5/7/2016 - I went back to the engineering team directly for corrections on what did/did not get carried over, and it's even more "clean sheet" than I thought - some of my comments here were to quick and in error - please read the full comment thread - and yes, I make mistakes :-) ]

[UPDATE - this response was wrong] @Moez - thanks for the comment! Dedupe was an omission (or insufficiently clear) on my part. There is dedupe for both block and file in the GA release (was GA on 5/2/2016). Indeed, there's also compression. However, they are both post-process implementations. The comment about compression coming is that there will be an inline compression implementation.

[UPDATE - this response was wrong] @Mark - thanks - see my comment above. Dedupe exists - but it is post-process (improved materially over the VNX2 implementation, but still post-process) for NAS and block. Double checking on the date for inline compression. We've found that when customers are looking at the $/GB economics, they (understandably) will consider inline implementations of data reduction (both compression and dedupe) in their economic model, but not post-process. We're expecting 1.5-2x from compression, and are willing to put our money where our mouth is (in customers pricing models).

[UPDATE - this response was wrong] @abj - thanks for the question - see above.

@PT - thanks - "Lifetime" means for the service life of the platform (EOSL). That means that customes can count on maintenance/support costs not going up, and that we are responsible for the full lifetime of the SSDs for wear and tear (WPD/wear-levelling)

@Frank - re: online data in place storage processor upgrades, stay tuned. We have this cracked on VMAX all-flash and you'll see the same on Unity. Re flash abatement see above. We're confident, and putting our money where our mouth is (lifetime media guarantee). Write abatement isn't as strong as in XtremIO for example, but there's now a lot of data coming in on WPD, wear leveling and write abatement effects (including the effect of caching, both in DRAM, and by leveraging high WPD media as a primary write stage).

[Mark Burgess]

Hi Chad,

Thanks for the feedback.

We have been told that de-dupe has not been carried forward from the VNX2 and we have looked in the Unity VSA and there is no sign of either post-process compression or de-dupe - is it definitely there?

Are we likely to see inline de-dupe in the not too distant future?

Quite important I would have thought.

Many thanks as always
Mark

[sk]

Nice to hear that the post-process dedup/compression is still there - documentation doesn't really mention it at all!

Is the inline compression coming for AllFlash only or Hybrid too?
And can you check the timeline again, is it really on Q2?

thanks,
sk

[Sami]

Hi,
I'am seeing mixed replies now.
You stated that Unity has post-process deduplication, but in the Ask the Expert forums they replied that there is no deplication at all.
https://community.emc.com/message/936184#936184

[Chad Sakac]

All - let me triple check - I could very well be wrong. Will go straight to the source for the authoritative answer.

[Chad Sakac]

@Mark, @SK, @Sami - thanks for staying on me for the final, correct, authoritative answer.

1) YOU ARE RIGHT - I went to the engineering team directly, and the decision was made NOT to carry over the block dedupe in VNX2.

2) ILC (inline compression) is 2H 2016, not Q2 2016.

3) They are prioritizing ILC over dedupe right now, but lots of internal dialog how to bring the best cost efficiency we can bring to bear (including dedupe). Will bring that dialog to a close and share as much as I can.

Thanks - and highlights the power of the community to get to the right answer fast (and that I'm FAR, FAR from perfect!)

[Brock Solutions]

Where is the Cloud integration?
Doesn't EMC own TwinStrata? Seems like a no-brainer, especially with EMC/DELL's new Cloud initiatives.

[Patrick]

Hi,

I've been working the the UnityVSA lately and have some feedback / questions:

1) Analytics - they suck unless you use the CLI (and read the 471 page manual). It seems like EMC is pushing Unity customers (or vVNX/VNXe3200) to buy EMC Storage Analytics w/vROps. That means I have to rewrite all my scripts and/or purchase additional software and learn VROps! Also as far as I know the REST API doesn't support analytics, just provisioning/configuration. Analyzer was a pain to use sometimes, but it gave me everything I need, not dumbed down stats. For a midrange array, I would expect all of the analytics available via EUMCLI to be available in Unisphere.

2) What about migration from CX4/VNX/VNX2 (and Celerra / VNX for File)? Does Unity support Mirrorview, SANCopy, or Celerra Replicator? Does it support Virtual Data movers? I doubt one could do a data in place upgrade from VNX -> Unity, so data migration would be the only option. Would I have to purchase RecoverPoint to migrate LUNs from CX4/VNX/VNX2 to Unity? Does Powerpath work with Unity?
It seems like Unity is geared towards VMware, making S-vMotion the obvious choice for migration - but what about shares and stand alone server LUNs?

3) Is there a UPS in the Unity system? I've always liked the idea of the Celerra/VNX for file having data movers dedicated for NAS, 1-2 control stations, and storage processors. They performed great and never let me down. More complex yes, but always predictable.

I'm currently using UnityVSA to learn the new operating environment, but I doubt I'll buy Unity until all the issues are worked out. Rewriting all my scripts, and migrating NAS I see as a big pain. As far as VMware environments go, I think it could be a good fit.

I'm not an EMC hater, I've been a loyal customer for 10 years, but this new surprise platform seems to be (atleast, at launch) a beefed up VNXe3200....

[Brian Henderson]

Patrick,
On behalf of the product team, thank you for trying out our Virtual Storage Appliance and sharing your feedback on Chad’s blog.
We’d much rather hear the feedback than have complaints go unsaid and not have a chance to address it and/or fix it - so thank you for that.

I will respond to each of your points below:

1. Analytics
2. Migration
3. UPS

Analytics

Unisphere has dozens of performance metrics available. These charts can be manipulated as desired, such as breaking down metrics by SP or read/write, viewing certain timeframes, and exporting to a CSV file. In order to leverage metrics via CLI, there is no need to read the entire manual. There are only two commands needed to view metrics, one for historical and one for real-time:

uemcli /metrics/value/hist
uemcli /metrics/value/rt

REST API can be leveraged not only for provisioning and configurations. In addition, REST API can be used to extract metrics information such as the ones mentioned above.
The EMC Storage Analytics plugin for Unity is included for free with the system. This allows customers to leverage vRealize Operations Manager for additional topology, metrics, and analytics information. No scripts need to be rewritten as everything is done automatically by the plugin. ESA provides additional information to complement what’s already available on the array (see above) for VMware environments, but is not required to view performance information.

Migration

There are a number of options for migrating both block and file data to Unity. For block, Unity supports SAN Copy push from CLARiiON/VNX1/VNX2. Unity also has integration/support with a number of migration tools and software, including RecoverPoint, PPME, and VPLEX. For file, traditional tools like EMCopy, Robobcopy, and rsync are supported, and we also offer a solution from our select partner Datadobi. This migration software, DobiMiner, performs file system migrations to the Unity platform. DobiMiner is compatible with many different source EMC storage systems, including a set of third-party storage arrays. Lastly, for virtualized environments, RecoverPoint for VMs and Storage vMotion is also supported.

On the topic of replication, Unity uses NAS servers, which are similar to VDMs. A NAS Server, which is required before creating file systems, allows for multi-tenancy in that each contains its own distinct set of configuration information and file interfaces. Because each NAS Server is logically separate, it is possible to segregate access so that clients of one NAS Server are not able to access data on another NAS Server and vice versa. Each NAS Server may contain up to 10 interfaces and a variety of configuration information including naming services, sharing protocols, active directory domain settings, a UNIX directory service, user mapping configuration, data protection settings and more.

As far as replication is concerned, native block asynchronous and synchronous solutions are included. If you want to replicate block resources between Unity and VNX1/VNX2, RecoverPoint can be leveraged. For file, native asynchronous replication is included. File replication between Unity and Celerra/VNX1/VNX2 is not supported, as Unity uses an all new 64bit file system technology with new space efficient snapshots and increased limitations.

We have some really cool things coming soon to make migrations even better and more seamless.

UPS

There is no UPS unit with Unity, but you can certainly use a UPS of your choice with the array. Unity doesn’t use separate Standby Power Supplies, but instead leverages a Battery Back-up Unit (BBU) in each SP. The BBUs are designed to supply enough power to the enclosure temporarily, in order for the SPs to flush the write cache contents to the internal M.2 SSD, which is non-volatile. When power is restored, the M.2’s contents are restored to the SPs.
Since Unity is built from a redesigned OE, we are able to fully integrate hardware components. There is no longer a concept of “passive” hardware which would be consuming valuable rack space and power resources. Both the control stations and datamovers have been replaced with an updated File architecture that utilizes the SP hardware and completely integrated management stack. Unity utilizes an active/active controller model – both SPs can be used simultaneously so no dedicated standby hardware is required. The peer SP acts as a hot standby, which actively services I/O but is also ready to take over additional resources if necessary.
We think we have done something quite different here and I’m glad you were able to try the product for yourself for free in advance of a purchase. The entire product team has seen the post and we’re making sure we are taking your feedback into consideration for our upcoming releases.

Thank you,
Brian Henderson
and the Unity Product Team