“What is SDS?” battle card/decoder

Perhaps an (intentionally) inflammatory title – this topic rages as furiously as the SDN battles, and the AFA wars… Hey at least things aren’t boring :-)   I still think we could have better things to argue about – so many things are more important (like let’s just abstract as much as we can, get used to programmatic APIs, and get on to the business of building new apps with new data :-)

The bulk of this I’ve seen actually has been between VMware and EMC field teams (because both VMware and EMC have leading products in the market here), but also read an interesting post from the Openstack Cinder PTL John Griffith on “The problem of SDS layers under Cinder” which I would encourage you to read here.

My quick comment to John is that: a) I disagree with the meta point; b) I agree that if you’re focused on Openstack exclusively, you don’t need an SDS abstraction under Cinder – after all that’s what cinder does.   In the case of b) you just need a good vendor driver, and the vendors can innovate around Cinder and contribute to Cinder.  

That’s actually why EMC contributes (and will continue to provide) direct Cinder drivers for the storage platorms we make, and also a Cinder driver for ViPR.

BTW – the always awesome Jim Ruddy also commented on John’s post here.

The confusion centers around “what is your point of view?”

Here’s how I look at it…

In the realm of “SDS control/policy/abstraction”:

• EMC ViPR Controller = SDS control/abstraction that is open (VMware, Openstack, Physical, Microsoft, etc) - it can plug into Cinder, SPBM and others.  You don’t need ViPR if your hypervisors, hosts and northbound management tools are homogenous.  ViPR Controller is also is an abstraction layer that is common for different types of storage block, NAS, HDFS, object and other protocol types – so if you find yourself with some transactional stuff, some general purpose NAS stuff, some HDFS stuff, some object stuff – it’s handy.
• Cinder = SDS control/abstraction that is Openstack specific.   You don’t need anything beyond Cinder if you are homogenous Openstack, or are OK with using different approaches across your heterogenous stack.   Further, you are ok with using a different abstraction model for NAS (Manila) and object (Swift). 
• VMware SPBM = SDS control/abstraction that is VMware specific.   You don’t need anything beyond VMware SPBM and the vCloud APIs if you are homogenous VMware, or are OK with using different approaches across your heterogenous stack.   Further, SPBM is centered around the idea that “consumer of storage” is always a “VM”.   You will need another abstraction for an external NAS pool that VMs might access, or an HDFS pool that might be consumed by VMs.

In the realm of “SDS data planes/data services”:

• ViPR Data Services (ScaleIO, ViPR Object/HDFS stack) = SDS data services that are open (can be used for many use cases via open protocols).
• Redhat (Gluster and Ceph) = SDS data services that are open (can be used for many use cases via open protocols)
• VSAN = SDS data service that is VMware specific (can be used only to store VMs running on vSphere)

BTW – what I DO NOT think applies as “SDS”:

• An external hardware array that implements APIs well.   Doing an API well is something that at this point is fundamental (and I’m trying to drive hard across everything I can).  
• Note: If this “hardware appliance” gets subsequently implemented as a software-only (implemented on commodity off-the-shelf hardware) variation (think Ontap Edge, Nexenta, or VNX Liberty, or vONEFS) I would call that “SDS Data Plane/Data Services”,
• An external hardware array that can have other storage behind it (aka “storage virtualization”).
• Note: If this hardware gets subsequently implemented as a software-only (implemented on commodity off-the-shelf hardware) variation (think VPLEX/VE) I would call that “SDS Data Plane/Data Services”,

This is a complex ecosystem, and intrinsically everyone (me included) tend to focus/bias on our domains of focus – but I think it’s “off” for anyone to claim that something “is/isn’t” SDS (which BTW, John didn’t – he was expressing his view that “SDS Control under Cinder” doesn’t make sense from a “homogenous” Openstack viewpoint (which I agree with – see bullet list above).

Comments

[Chuck Hollis]

Hi Chad -- Great discussion!

I have to say, I find myself continually unsatisfied by the definitions of software-defined storage that most of us are using.

Most of them center around a programmatic exposition of pre-defined storage capabilities, e.g. here’s your gold, silver, bronze, etc. — what do you want? Maybe we’ve changed the interface, but we really haven’t changed the underlying model.

Maybe it’s the fact I work for VMware, but we’re pushing for a far richer model: storage subsystems expose potential capabilities, applications expose a policy-based manifest of what they need, and the SDS layer dynamically composes a storage container and associated data services precisely aligned on application boundaries.

Change the policy, the SDS layer either instructs the storage to adjust, or transparently moves the workload to a new location where the new conditions can be met.

The difference might be described as fast food where everything is made far in advance vs. a sit-down restaurant where your meal is cooked to order.

As I’m not all that familiar with the direction of Cinder and related projects, I’m not sure whether this is part of the broader thinking or not. From what I understand, Cinder does bring a lot to the table in terms of consistent access to underlying resources (a good thing) but seems entrenched in a static model.

Time will tell whether people demand a more dynamic model (such as we use with compute) or not.

Best regards!

-- Chuck

[Keith Norbie]

Chad,

Love the post but slight take issue. In the overall context of SDS I do think the objective we're trying to reach is moving orgs from hardware defined to software defined on the same way servers used to be hardware defined. (remember BMR, ugh) IMO SDS can be defined for Commercial and SMB with a single array or a small amount of similar arrays. Tintri gave use one of the 1st arrays that did this. Abstracting to a non-storage specific output that is VM centric was a good start. I would argue that we just need to get beyond these barbaric means of serving storage through LUNs, etc. 2nd Platform is a good mess to address (and I'm having a ton of fun doing that with EMC). IMO 3rd platform SDS become interesting due to programmatic interfaces. I love that an Exabyte of ATMOS in run by 4 policies (no SMB, no NFS, no LUNs). THAT's SDS too :)

[Chad Sakac]

@Chuck - thanks for the comment, and it's great to hear from you Chuck! I don't think that you're off on an island (or that it's a uniquely VMware view). IMO, abstraction, automation and policy is good (period!), but that ultimately the infrastructure itself should adapt to policy demands, and SLO/SLA adherence.

That latter part is not an easy technical problem (and I think we're a ways away from that - but you can see the industry aiming at that target, from VM-level policy, through to QoS mechanism etc). I know we're both working on it!

@Keith - I agree that SDS has a key element of "policy" and similar to Chuck's point - when you can do it at the unit that the application/use case demands, it's good. Tintri, and VSAN with VM-level operation are examples. I'm very much looking forward to VVOLs which will (I think) move the industry as a whole forward. It's notable that not all application/use cases consider the "VM" as the desirable unit - for a lot of analytics workloads, an HDFS object is the "unit", or a KV store pair as an example. BUT - the meta point is valid: LUNs and filesystems are not (for many use cases) the "desirable unit". I stick to my guns though - while that's GOOD, the point is about programmability (with the unit of policy/automation being the right one for the given use case). I would also argue that if that programmability is "tied" to a given hardware platform, I struggle to call it "SDS"...

The debate rages!

Thanks for the input guys!