Cloud storage – what the hell is EMC building?

Just got back from 2 weeks in Europe with customers of every shape and size.  One thing that was clear to me was that we need to be more clear about our cloud storage strategy – as many of them were struggling with issues and didn’t know what we could do to help them.   Then again, I’m not sure that I will help make things more clear with this post - one thing I’m never is “talking point concise” :-)

So – what is our cloud storage strategy? 

First of all, it’s very important that cloud doesn’t imply “internet and public” by definition.   It does imply these things:

1. usage models (large aggregate pools, oversubscribed – not “hard provisioned/thick”)
2. consumption and economic models (pay-as-you go)
3. management models (zero-touch from a user perspective, with very “cross silo” tools from an administrative perspective).  

These core ideas and principles apply all infrastructure services (IaaS) models – including both compute and storage.   They also imply whether those services are constructed and maintained by an enterprise in within/across its walls as a purely internal service (presumably higher SLAs and security than public cloud), or by a service provide (presumably higher degrees of scale and oversubscription and therefore more efficient).

If this sounds “future speak” – it’s NOT.   EVERY customer I talked to was trying to build compute and storage self-service internal IT services.   They may call them different things, but when you describe it like the above and ask: “are you trying to do something like this?” the answer was always “YES!”.

As any reader of Chuck’s blog  (here, here, and here) knows – EMC’s view is that both Internal Clouds and External cloud models are winners for enterprise customers and service providers, and once we crack the nut of open-inter cloud standards and federation – whole new technology and economic vistas open up for our customers.   Even before that – well – if you’re building a new application/infrastructure/datacenter – customers tend to be thinking on these lines.

But… I leave the high-level thinking to Chuck (he does a much better job of describing this at a high level than I do), I want to spell out the implications on this in perhaps a lower level of detail.    I also sketched it out for a couple customers, and thought the diagrams are perhaps useful to make this simple to comprehend.

If you’re interested – read on…

The implications on storage infrastructure of these internal/external cloud models profound.   Think of it this way…  The vast majority of storage users in enterprises today have a provisioning model where the first step is “tell us whether you want 250GB or 500GB, whether it’s SAN or NAS, and the protection level – then wait a couple weeks as we process the request”.   Of course, to fulfill the request, they purchased a wad of storage a year ago.   How much did they buy?   More than they needed – because god forbid they err with not enough (and in doing that ensure that they err completely in the opposite direction!).   And, of course, it’s generally pretty “thickly” provisioned – and even if it is thin, it’s doled out and managed app by app, so the “pools” tend not to be too wide.

Not very cloud-like.  

While nothing we have today is perfect (I don’t think any one thing is), the architecture of what we DO have is designed for this purpose.   Here it is in a nutshell.

First, we’ll discuss Compute as a service (in other words, the thing you’re using as a service is a VM) and it’s storage models.   Next, we’ll discuss Storage as a service (in other words, the thing you’re using as a service is GB of storage).

Compute as a service and storage: 

Let’s say you want to construct a “compute as a service” Internal (or external) cloud – in other words: a self-service, pay as you go, hyper-efficient datacenter, and one that could support everything from your test and development, all the way through Exchange 2010 and Sharepoint deployments and continuing to your Oracle 11i/11g stack and SAP landscapes.  

Who wants to build a datacenter on this model?  Pretty well every customer (names everyone knows) I talked to over the last 2 weeks (and for the 6 months prior to that).  

They are worried about the implications on their IT organizations.   They are worried about management tools for end-to-end compliance, remediation and root cause.   But – they know it’s the right model.   They are parking their mainframes and HP-UX/AIX stuff to the side, and saying: “ok, not touching those – but going fwd, this is how we will build the x86 compute environment…”

You would of course want abstraction at every level.  Here’s a quick diagram that shows the core architectural model, what the customer is looking for, and the requirements implied at each layer in the stack.

 

Note a couple things:

1) Where does the multi-tenancy and VM distribution policy reside?   By definition at the vSphere layer and above.   Everyone who has created a service on this - think Terremark, or EMC’s own at www.emccis.com – have built self-provisioning, multitenant, and chargeback layers on top of VMware.    Today, we’re all building our own “homebrew portals” aligning to the vCloud APIs.  You don’t need to be an insider to be in on the not-so-super-secret that VMware is working on making a “out of the box” layer there to make it even easier.

BTW – I can’t be emphatic enough about this: EMC has stood up our compute and cloud storage services to act as a PROOF POINT of the end-to-end technology, not to compete with the service providers/telcos.  There’s always so much talking at the cloudy-level about this stuff, that many think it’s “future tech”, not “now tech”, or some of the slower-moving service providers want to see how the whole thing works together.  That’s what www.emccis.com is all about.  It is real , it is something you can sign up and use, but it is based out of two datacenters in the Americas only.

2) You need to have a scale-out compute layer.   This is provided by vSphere 4.   Now, note that there is a baseline in the industry that this part of the stack needs to support a target economic model (measured in cents/VM/hour).   Extra services, extra security, extra compliance can allow for higher $/VM/hour, but think about that model (and look at the vCloud partners, or the EMC Cloud Infrastructure Services portal for details).

3) This compute layer will always have a degree of oversubscription.  In an enterprise, this may be a little.   In a service provider, it is generally greater.   It also means that things that can control quality of service  (in vSphere, this is DRS) are very important.

4) What about the implications for storage?  First, note that it needs to support a transactional workload.   Second, it needs to have a “go wide, and be thin and oversubscribed model”.   The economic model is also very specific – think “cents/GB/month + cents/100IOps/month”.    Ideally this would scale out in a very abstract way, with the infrastructure being able to start small, and grow big, being able to add IOps, port, MBps, and gigabytes all as independent properties – ergo a big pool you can add to as you need. 

What about protocol models?   Well, while I drew SAN/NAS/Object with a big question mark, the protocol choice is more basic.  

• Until future VMware releases – native object models are definitively out, as VMware expects only block or NAS devices.    Also, object storage is not a fit in general for transactional workloads.  Object can be used for templates and other low-IO workloads, when front-ended with “gateways” that present them as block or NAS devices – but this isn’t ideal. 
• Block models can have horizontal “scale out” models, as a LUN device can be actually presented by many ports across many horizontally scaled controllers with MPIO driving parallelism.     You can have a block array that can start small, and dynamically add ports, brains, disks all as independent variables.  Note that SAN doesn’t by definition mean “FC SAN” – iSCSI, FC, FCoE are all options.   Examples of this range from EMC’s V-Max, IBM XIV, 3PAR, heck, even Dell/EqualLogic and HP/Lefthand (though those last two are generally not designed for the scale we’re discussing though I’m sure they would disagree).   The relative merits of one versus another I’ll leave to sales and competitive teams, but I wanted to point out examples of the model.   Some VM-placement logic is needed as there are core vSphere VM per LUN scaling considerations.   One of the goals of the next major vSphere release will lift this element of design (see TA3220 or SS5140 and look under the vStorage APIs for Array integration – specifically the hardware-offloaded locking, or Atomic Test and Set APIs/array offloads).

I also want to point out – there is not a long list of vendors really doing this.   EMC certainly is.   We are shipping V-Max in volume exactly for this use case, and our scale-out block storage, that can meet the economic models noted in the diagram is here and now. 

• NAS has a couple extra considerations here.   With vSphere’s use of NFSv3, it means a single connection per datastore.  This means that “horizontal scale-out” NAS is impossible for now in the use case of storage behind vSphere.   You can use NAS of course, but can’t horizontally scale the storage, you need to build into the operational model (and potentially the provisioning portal logic) that there are multiple NFS servers/arrays.   An example would be that you would build in building blocks where a sets of large 32-node vSphere clusters are backended by a two (or more) separate Celerras – as shown in the diagram below.  For some this isn’t a showstopper, for others it is.     You can also use this model with non-scale out block storage arrays.   The downside is that excess capacity/performance across the backend platforms cannot be easily used across the horizontally scaled compute layer.   It’s a bit of a purist architectural point, as some degree of “VM placement” logic needs to be built into the provisioning front end, and it’s simply a case of adding storage logic to that as well.   BUT, it’s not ideal, and breaks the idea of “horizontal scaling of all elements”. 

Rest assured, EMC and others (of course I’m talking about my friends and colleagues at NetApp) are working hand in hand driving the pNFS efforts, and to accelerate scale-out NAS models behind future vSphere generations, focusing on pNFS client development with VMware and scale-out pNFS NAS models.

 

Storage as a service: 

Let’s say you want to construct a “storage as a service” Internal cloud (or external cloud) – in other words: a self-service, pay as you go, GB/month model.  

Who wants to build a storage on this model?  not as many in number as the compute use case (but this is an emergent area, and growing fast), but the people that do, REALLY need it, and need it in huge volume.   The story of Facebook using 25TB of daily data ingest, and needing to move away from traditional NAS models has gotten wide circulation   If you’re in that market, your choice is essentially: 1) use Amazon S3; 2) home-brew your own (what Facebook did) using various open source bits and pieces – in their case Haystack; 3) use EMC Atmos.    If you do option 1 it’s good as you startup, but there’s never the choice of taking it and running it yourself.    Conversely, the number of customers who can do option 2 is pretty small – it better be pretty close to your core business.   Our proposition on option 3 is: “start in our cloud storage service if you want, and if you later you want to create your own cloud storage service, we can give you the technology as a turn key option”.

BTW – if you want more info on Facebook, good places to understand (and their story highlights why this is a new need, needing new architectural models), check it out on this excellent Facebook Techtalk:

http://www.bestechvideos.com/2009/06/08/facebook-tech-talks-haystack

Now, note that this is similar in the usage/economic model to the compute as a service model (they are both scale out, pay as you go, zero touch models), but architecturally different.   Like the compute as a service example, you would of course want abstraction at every level, but there are less levels.  Here’s a quick diagram that shows the core architectural model, what the customer is looking for, and the requirements implied at each layer in the stack.

 

Note a couple things:

1) Where does the multi-tenancy and VM distribution policy reside?   By definition at it has to be embedded in the storage model itself – very different than the compute as a service case.   Since the storage must present namespaces that are isolated from each audience, you can’t just “hide” this in the provisioning front end.    Note that this doesn’t just mean “multitenancy”, but other policy things like geo-distribution of content.  To understand this better - geographic distribution is good in some cases (example use case is YouTube video distribution), and absolutely not allowed in others (example is some European Union rules require that the information for users is stored in the same country as the user themselves and in NO OTHERS).

2) The presentation model that seems to be the most popular in these use cases seems to be the object REST/SOAP model.   It makes it very easy to use when leveraging the content to build web-based applications (exactly the Facebook example)

3) Note that the storage model must do EXTREME horizontal scaling here – much, much more than in the case of storage supporting Compute as a Service.   These storage as a service models are almost always built on models that in essence glom together commodity servers with very dense DAS/JBOD storage configurations.

4) What about the implications for storage?  First, note that it is a different workload than the transactional workload of the compute as a service scenario.   Second, the “object” nature is CORE in the model, not simply a question of presentation.   The economic model is also very specific – think “cents/GB/month”.   

I also want to point out – there is not a long list of people really doing this.   EMC certainly is.   We are shipping Atmos in volume exactly for this use case, and our scale-out object storage that can meet the economic models noted in the diagram is here and now. 

Conclusion:

I hope this helps explain what we’re doing and why.  The key thing to takeaway is that unlike a lot of talk, this is shipping stuff.   

Interested in a self-provisioning compute layer in your datacenter, where VMs cost tens of cents per hour, and the storage that supports them costs tens of cents/GB per month + cents/millions of IO usage rates?   We can help you today.  

Interested in a self-provisioned object storage model for your internal web apps, or building an external web app?   We can help you today.

   We have constructed it for service providers, and the EMC Cloud Infrastructure Services portal is up as a proof point as much as actually being a service.

So… what’s next?

1. Blending and blurring of the lines – object storage will move up to support transactional workloads better, pNFS and scale out NAS will come, and block and NAS will get some metadata structures bolted on.
2. Federation at scale in the Block (soon) and NAS (less soon) in the compute as a service – the object storage as a service model does this already

This is a SMALL part of what we’re doing (as much is happening on the management, security, and backup/recovery parts of EMC around these new models) – but those are posts for another day.

Comments

[Donald Poorman]

Hey Chad - The one thing www.emccis.com is vague about is if there will be Internet bandwidth usage costs thrown into the model. Amazon currently does this for both S3 and EC2, mainly because it is a VERY profitable add on piece for providers. I even signed up for the service but couldn't see if that would be a factor in the final cost; EMC might be leaving money on the table if they ignore this.

Great post!

DP

[Gregory Matthews]

Chad, thanks for putting real clarity to these concepts. Very helpful.

-Greg

[Jeff Aaron]

Great overview, Chad. One thing to remember is that all of these models are reliant on adequate network performance. As storage moves into the cloud, it is more important than ever that the WAN has enough capacity and suitable throughput to support these initiatives.

That is one of the reasons why EMC has partnered with WAN optimization vendors, like Silver Peak.

[Chad Sakac]

@Donald - thanks. Variable rate items in this cloud use cases is a topic of much discussion (what's the right balance?)