[updated June 24th, 11:10am ET – minor typo fixes, one technical catch, fixed links]
DockerCon is going on today and tomorrow – and no better time to put out some new great tools for people using Containers.
This will be a 3 part post – and timed intentionally with DockerCon. Why? Well – containers are indeed the buzz du jour, and while the ecosystem is still very vibrant (far from settled, with lots of battles being fought), DockerCon has become a critical time for the ecosystem to get together and collaborate.
Containers and persistence – what’s the scoop?
A lot of strange new ideas for people used to NAS, SAN, and VVOLS :-)
Object stores tend to be the dominant persistence model for people building apps using containers – but there is emergent work around Flocker and other things for other forms of persistence (parts 2 and part 3).
Frankly, developers generally they don’t think about “persistence” beyond the data layer. “Data Layer” is not what people think of as “storage”, but more like “RDBMS” in traditional apps – where an app writes something with an expectation it will persist. In modern “12-factor” apps (sometimes called “platform 3” apps), the data layer is a “polyglot persistence” layer (which is a terrible way to say “multiple languages of persistence”) – consisting of a composite of in-memory DBs, Key-Value stores, scale-out NoSQL models, MapReduce. These all which in turn depend on transactional storage models – more on that in part 2 and 3. The other form of “persistence” developers use areobject stores.
So – what’s the news? Well, Elastic Cloud Storage (ECS) 2.0 is out now, and available for free and frictionless download – I would encourage you to ignore the rest of my post, download it, and try it for yourself here if you like binaries. http://www.emc.com/getECS. It’s also in the Docker Hub if you prefer the Docker marketplace. If you want to be like the cool kids and love a git clone, the github repo is here (for single and multiple node configurations) Give it a whirl :-)
Here at EMC, we’re really getting into this “make access to our software IP easy” thing, and there is a pattern: /getECS, /getScaleIO, /getIsilon - “get it?” :-) I’m trying to guide our sales teams to back away slowly – this isn’t a case of “let me give you a huge powerpoint”, but rather – guide a customer on access, use, and answering questions.
Elastic Cloud Storage is EMC’s premiere, modern and “designed from the ground up” object store.
In a sense, we’ve been at the Object storage business for a looong time. Way back in the day (2008) – EMC released the first release of Atmos, which was our first geo-distributed, cloud scale object store. The architects cut their teeth on OceanStore at Berkley. Atmos is now used around the world with many, many PB-EB scale deployments.
It’s funny – while so much talk in the storage world centers around the battle-ground for “platform 2 transactional storage (think of all the poop flinging in the AFA battles) – the world of object stores is more quiet. People just use them without fighting. I think it’s because there’s less revenue being fought over.
That said – object stores are part of the fabric of our lives every day – and underpin all modern apps we use, whether they use Atmos, AWS S3, the Azure blob store, a Swift or Ceph variant.
This stuff is all around us all the time - in fact, I would wager that many of us, at some point in the day, dear reader, you will use an Atmos-based object store without even knowing it.
While I wouldn’t call it “an object store”, Centera is an object store architecturally It’s an object store designed for compliance use cases, and the CAS API .
Now, EMC has been at the world of Object Stores for a long time – but when you build something, you learn a lot – and sometimes those lessons are hard (because they are deeply wrapped into the architecture).
In 2011, Amitabh Shrivistava and Manuvir Das joined EMC from Microsoft – where they learnt hard lessons in helping build the earliest forms of Azure. Based on that experience – they wanted to build an open storage policy abstractor/automation tool (which became ViPR and the open-source trunk, CoprHD), and a next-generation geo-distributed object store – Elastic Cloud Storage.
- Scales out the ying-yang. Think many billions of objects, easily.
- Performs great across a broad band of object sizes – and at huge scale. With other object store implementations - object ingest and read/write rates vary wildly from small objects to massive blobs, and also wildly vary with scale.
- Rich Geo-dispersion – which is important for cost and scaling. Most modern object stores use an erasure coding (vs. mirroring) approach. ECS uses a technique which asymptotically approaches ~1.5x efficiency (1.6x at 6 sites)
- Strong consistency. Object stores (unlike NAS and block POSIX storage stacks that consider consistency an inviolable design requirement) have varying forms of consistency. Loose consistency means that the app developer would need to build logic to “check” that they actually got what they asked for. Complete consistency is what most people expect – but is generally a trade-off with geo-distribution. ECS 2.0 has a strong consistency model (you always get what you ask for), but also implements geo-distribution via a novel approach.
- Open API layer. With a really modular northbound API layer that is very decoupled from the chunk-management layer, ECS can support a broad set of northbound APIs. ECS supports a superset of S3, Atmos, Swift APIs, and a subset (still missing compliance) of the Centera CAS APIs today. In fact, with ECS 2.0, we really think we can cover the vast, vast majority of Atmos use cases with ECS (Atmos “3.0” is in effect ECS - “Atmos is dead, long live Atmos” :-) But, ECS can also support HDFS via a custom libHDFS, and in the future, we will add NFS for people who want a file handle for and object occassionally. This isn’t a different “pool” on the same thing – each of the protocols accesses the same content, without moving or copying it – handy. [added 6/24 - one note – while not a inherent limitation, the current access layer of the stack enables object + HDFS access to the same object, but not simultaneous through two object APIs like Atmos API and S3 API simultaneusly – we have a customer looking for that, and are evaluating adding that capability].
- ECS Built on a container-driven model, with a unique stateful container management bit of code. This is a neat little thing. Things like Kubernetes, Mesos, and the CF Run-time management layer are all “cluster managers” (managing the container/VM abstraction units, monitoring health, and taking action). However, most of these don’t have a rich set of semantics for the relationships/dependencies of containers. Ergo they go “hey, this container/VM is dead… oh, well, restart it wherever the resource manager says there’s balanced capacity”. For object stores – that doesn’t work, because state matters – you don’t want to restart part of the object store on the same server/rack/data center as another part based on the geo-dispersion/erasure coding logic. More on this tomorrow in Part 2 and Part 3 - it’s interesting as the container ecosystem starts to tackle persistent use cases in addition to the ephemeral things they focus on today, they are starting to add not only persistence, but tackling questions of rich hand-shakes/state management in the cluster managers. Frankly, I’m pushing that we should open-source some of the cool stuff that we’ve built here – and use it to contribute to the cluster manager ecosystem.
BTW – a little side-bar on that last bullet (our investment into container and container management) from Manuvir…
“EMC bet on containers (and more specifically Docker) before it was cool :-) - by building ECS from the ground up on Docker 0.1 and trusting that Docker would come around in time, which it has. Every box of ECS we ship has ECS running in Docker containers. Has been so from day one. The Dockerized ECS we are making available is exactly the same set of binaries we run inside our appliance. Docker is in the DNA of ECS. Which other storage system can claim this? Who else is on a path to run 100PB+ production data on Docker in 2015?"
I’ll add that containerization is at the heart of HYPERMAX on VMAX3 and the code that brings VNXe into such a powerful, small package.
ECS 1.0 and 1.1 were deployed at many customers, and 2.0 firms up a ton (including critical site failure behavior, rack topology handling, object/bucket auditing, reporting/management) – and we think this is ready for a ton to be thrown at it.
Now, I’m sure that many of you will be skeptical. Competitive Object Stores will surely disagree, but we think we have something superior to any of the object stacks out there. The joy of software being easily and freely available means you don’t need to take my word for it. Download. Give it a shot. It’s easy: www.emc.com/getECS
Now, ECS comes as full-software only form (and our most monstrous deployments tend to require that). Of course, when customers want support for the ECS software stack from EMC, they would license the software and pay support.
But, it also comes in appliance form (bundled with commodity hardware for an integrated software/hardware support model)*. This tends, in my experience – to be desired consumption model except for the truly hyper-scale customers. ALL customers WANT the flexibility of the hardware layer. MOST however, end up wanting a single accountable party to support the whole “hardware/software” stack.
*I agree Chris Mellor @ El Reg when he rails at the storage ecosystem “SDS-washing” things here (in fact I commented). That said – the concept of persistence software-only storage stacks is a real thing customers want, and VERY frequently strangely in the form of bundled appliances (I talked about the “why” of that based on my experience here).
The ECS appliance looks like the below. We’ll keep iterating on the hardware, but it already is an incredibly dense package.
Welcome to the world ECS 2.0! I can’t wait to see how customers will use you – and would love your feedback (positive/negative, bring it on! :-)
Post #2 (Containers and transactional persistence, Flocker, RexRay, ScaleIO) and #3 (more on Caspian) I’ll post tomorrow…