A… Fascinatingly frustrating ScaleIO update :-)

Five things I experienced last week triggered this post for me – which I have to confess, I’m writing a bit short tempered.

ONE: Tweet #1 (here) – a customer with a ScaleIO SDS cluster has a 2TB SSD fail…. And it rebuilds in 98 seconds. 98 seconds! It was a loud reminder of the incredible parallelism of ScaleIO – not only does it result in even relatively small numbers of ScaleIO SDS nodes using NAND being able to outperform (in bandwidth and in throughput) every storage array on the planet, but it also equals insanely fast rebuilds and rebalancing across the cluster… and with large numbers of nodes, the performance and rebuild/rebalance operations scale linearly.

TWO: Tweet #2 (here) – someone build a home-brew ScaleIO cluster with 3 server nodes… that could deliver north of 300K IOps (and this isn’t a maximum by any means – it’s possible to have a single SDS node (given enough SSDs/NVMe devices) do more than 300K – all by itself.

THREE: There’s a “people update” in ScaleIO land… We have a new ScaleIO Specialty SE leader – Joshua Petty, who wrote a great blog post series. His perspective having been at VMware for many years about the functional analogies of abstraction, pooling, consolidation, and agility/automation are particularly resonant with me because I remember the same “whammo” effect of ESX/vCenter so many years ago – where the only mistake people made was not getting started sooner.

FOUR: There’s a “product update” in ScaleIO land… We have updated the ScaleIO Ready Nodes with the latest ScaleIO software, and Automated Management Services (AMS) which makes discovery, deployment, configuration, monitoring/operations, and update/maintain functions with ScaleIO Ready Nodes simple and easy. AMS can be best thought of as an extension of the element management and orchestration layer for ScaleIO. AMS is included for free.

Sidebar: In HCI Rack Scale systems that build on top of ScaleIO (VxRack FLEX), we are expanding the HCI/Composable infrastructure layer M&O layer (which leverages AMS lower down in the stack) via something codenamed “Project Symphony” – a very cool open HCI-level M&O capability which looks at the whole system – inclusive of network, servers, nodes, storage, software, and system-level contextual relationships. Totally open, everything via REST APIs, goal is to make it the ultimate in programmable infrastructure and deepen integration with iDRAC, Open Manage Essentials, the Dell EMC and Cisco networking ecosytsems, and future goodness we are developing…. Lots to come on this front over this year.

In AMS, you can easily see the lower-level infrastructure (see example below), and faults are detected, reported (and can connect to ESRS).

Also – because while many customers love the full vSphere stack integration of VxRail and VxRack SDDC using vSAN, other customers love being able to have their SDS to be able to pool across vSphere clusters, direct performance/capacity independent of node capacities, or because they want a heterogeneous approach. Of course – the majority of the use cases with ScaleIO in enterprises remain supporting vSphere… So AMS has extended its capabilities to automate both the ScaleIO and ESX level operations during automated non-disruptive updates (screenshot below).

We’ve made all our Ready Nodes easy to order:

• Foolproof, configurable platforms, limited to validated and supported hardware
• A ton (600+) hardware configruations (up to 40 cores per server, from 64GB to 1.5TB of RAM, SSD options of all shapes/sizes).
• An easy-to-use, online sizing tool to help choose the right configuration (https://scaleio-sizer.emc.com/)
• Simple Online Solution Configurator templates ensure quick and accurate quoting

… and we’ve also made our Ready Nodes easy to deploy:

• Customer-downloadable deployment package for BIOS and firmware settings ensures the system is ready for the ScaleIO application
• Customers can get it here at the Dell EMC support portal.
• Solution implementation services are available from Dell EMC (MyQuotes p/n: PS-BAS-SIOIMP) and I would highly recommend ScaleIO: Design (MyQuotes p/n: PS-BAS-SIOADDES)There’s a “product update” in ScaleIO land… We have updated the ScaleIO Ready Nodes with the latest ScaleIO software, and Automated Management Services (AMS) which makes discovery, deployment, configuration, monitoring/operations, and update/maintain functions with ScaleIO Ready Nodes simple and easy. AMS can be best thought of as an extension of the element management and orchestration layer for ScaleIO. AMS is included for free.

FIVE: with all that great news, the last trigger – I got a punch in the face from a friendly customer :-)

I met with a customer who asked me: “how should I determine what architectural model for storage I should use?”

I said: “it’s actually pretty simple:

1. “Start by assuming that the majority of your x86 workloads can be supported by SDS/HCI models. Yes, there are things that aren’t good at/capable of – but that increasingly, those the workloads that demand those capabilities are the exceptions (by COUNT not VALUE). Start by solving for the rule, not the exception”. They looked shocked – and asked if the stance of the company leaning into SDS/HCI was “secret”. I pointed out that NOPE – we are loud and proud on this (read this post here).
2. Then they asked “how do I pick which transactional SDS is the way to go?” Answer, again, is SIMPLE. “Choose which (or both) of ScaleIO and vSAN are the best fit based on: a) SDS for a VMware only focus – and always HCI operational model” (vSAN); and/or b) an SDS that replaces a multipurpose SAN (including vSphere) where pooling, capacity/IOPS scaling, and sharing needs to be independent of clusters, operating systems and other factors (ScaleIO)”. Again – they were shocked that we are so clear. They had heard two very different stories from the teams they interact with. VMware team said “vSphere always = vSAN” (not accurate – there are absolutely vSphere use cases where ScaleIO can be the right chocie). The Dell EMC ScaleIO specialist team said “Enterprise = ScaleIO” (not accurate – there are absolutely enterprise use cases where vSAN is the right choice). Smack (facepalm). The Dell Technologies position is public (here).
3. Then they asked: “But why are there 3 ways to get these SDS models – and which one is right?” Answer, again, is SIMPLE. “Choose how you like to consume – for people who like tinkering and the most flexibility, they tend to go software only. For people who like that flexibility but with a little simplification and de-risking of the project, they tend to go software + hardware bundles (vSAN Ready Nodes and ScaleIO Ready Nodes). For customers who want turnkey outcomes and understand and accept that comes with letting go of ‘I can chose anything’ – they tend to go full HCI Appliances and Rack-Scale systems (VxRail, VxRack SDDC, VxRack FLEX).”
4. Then they asked: “… But then what should stay on external storage (VMAX, XtremIO, Unity, SC and the like) along with Converged Infrastructure (VxBlocks)?” Answer again, is simple: “Most customers have some workloads that need specific data services (SRDF-like requirements; very, very high performance snapshots/replicas/dedupe); have very dense capacity needs; or very dense IOps/bandwidth needs; or very, very tight latency jitter requirements (100’s of microseconds) – this is the domain of traditional storage”. This by the way is a large set of workloads – and as importantly – workloads that are incredibly high value and important to our customers – and we have the industries #1 portfolio there.”

That’s four simple bits of dialog that are a simple (but relatively complete) answer to a complex question – the only thing simpler are the words from a single product company :-).

It may look like a lot of words – it’s not. Read it out – it’s about a minute of dialog.

The punch in the face was that the customer said next: “That’s good, and that’s clear… but it is not what your sales teams are saying.  They are pushing traditional approaches, array refreshes, and trying to marginalize SDS models, and the people talking about SDS choices are fighting about which one is right.

Look – SDS (vSAN/ScaleIO as examples) are NOT a panacea – and they aren’t always lower capital cost than external storage, though they are almost always simpler to operate, scale, automate, and ALWAYS simpler to upgrade and automate. BTW – the same applies to their HCI consumption options.

..And, of course the “right answer” to “what storage is right” for a given workload or customer can absolutely be the external storage arrays that people know and love (for all the reasons I stated above).

So – why am I frustrated?

I’m frustrated because change takes time, and like all waves of change (technological or otherwise) – it is people’s own fears that tend to hold us all back.

The biggest barrier of SDS and HCI adoption at this point is people’s fear, lag to adopt new approaches – and sales people who feel threatened (but shouldn’t).

Between ScaleIO and vSAN (and the HCI Appliances and Rack-Scale – we have a very compelling SDS portfolio for transactional workloads – and Dell EMC and VMware are neutral about whether the answer is one, the other, or as is pretty common – BOTH. Likewise, while we have a point of view – we know that every customer is a little different, and for most, they will be using SDS and traditional storage models together for years to come.

Advice from the heart: when things are changing, the most dangerous move is to stay still, to “stick with what you know”.

Remember – you can always put ScaleIO through its paces and come to your own conclusions. The bits are available for download here: http://www.dellemc.com/getscaleio

Are you a ScaleIO customer (software only, ScaleIO Ready Nodes, or VxRack FLEX) – and if so, how is it going?

Comments

[Andrew Carr]

Hi Chad I don't want to add to your frustration but when we talk about HCI we should also talk about XC Nutanix and Azure Stack.

Positioning the whole portfolio is a great neutral, open conversation to have with a customer.

[Howard Marks (@DeepStorageNet)]

Chad,

I know it was a customer tweet so you have some latitude but I have a sneaking suspicion that 2TB SSD wasn't nearly full. Rebuilding 2TB in 98sec is 20GB or 160Gbps.

If that was a full SSD it would have to be a large cluster with 40Gbps Ethernet to rebuild that fast without bottlenecking on the network or PCIe bus within a node.

It's a great line but you know the devil is in the details.

[Josh G]

I have been equally frustrated as a customer; because of Legacy EMC sales staff that ONLY want to peddle VMAX and Unity I have found myself only dealing with the specialists. The fact of the matter is that no matter what you do, some legacy sales staff will not adapt and get with the program. It is time to trim these older/un-adaptable individuals from the company and bring on fresh talent. I don't care how much VMAX they still sell; they are causing more harm than good.

[David Holmes]

Disclosure - DellEMC Employee

This is a great post. I've seen the same in the field as well. The key IMHO is to address the "if this then that" mentality. e.g. "if VMWare then (always) VSAN, if Enterprise Block then (always) VMax, if unstructured then (always) Isilon etc. etc." You have a great way of articulating the portfolio so that you don't back yourself into a corner. If more people started sentences with "assume that" and "choose ... based on" they would then have the latitude to help customers make the right choices....

[Kyle]

Chad,

Former team member and now competitor, so I hope you don't see this as a troll...I am genuinely interested. It always seemed like ScaleIO was missing the ability to monitor latency and it still seems visibly missing.

Three legs of storage experience: IOPs, Latency, Bandwidth...some of the posted IOPS are impressive, no question, but determining if it is a fit for customer architectures without the app experience in latency, kinda not really a full picture.

Curious how customers are navigating these differing architectures without all the visibility of latency as it relates to app delivery...perhaps I missed an update (honestly do not follow it much) where this is now in the management experience. If so, sorry for the spam! Hope you and the family are awesome!

KK

[VMGCloudOps]

@Howard Marks (@DeepStorageNet) - Actually, because of the high level of parallelism, that cluster need only be 8-10 nodes to realistically talk about rebuilding a 2TB SSD in under two minutes.

Remember that, during a rebuild, the entire protection domain participates in the rebuild operation, requiring only a fraction of that '160Gbps' to come from each node to execute the rebuild.

Also remember that any participating node has a certain quantity of the rebuild data, so each node is only sending a small portion of the data requiring re-protection.

If you have not seen this in action, you should ask for a demo from your DEMC account team, because it's staggering to watch a small number of nodes and SSDs rebuild at astounding rates.