2 extreme proof points from the NAS leader…

So – today, there were two releases that represent two totally opposite ends of the extreme when it comes to NAS (and suggests, perhaps, the way things will go over time).   Interesting to see how it represents where EMC is today (and hints at where we’re going) re the extreme entry-point, and extreme scaling point.

First – Dr. Martens (yup, the folks with the cool shoes) picks Iomega for their relatively modest (but still critical to them!) NAS and block storage for both primary and for helping their backup/recovery processes.

Dr. Martens has taken advantage of its Iomega StorCenter ix4-200d units to perform faster, flawless backups.

Dr. Martens also plans to leverage the Iomega StorCenter ix4-200d's robust, reliable performance and iSCSI connectivity to expand its reach within the organization. Additionally, the fact that the StorCenter ix4-200d is the first NAS server in its class to be VMware certified gives the company an easy and highly cost-effective replacement for its aging IBM SAN. "The Iomega StorCenter ix4-200d is exactly what I was looking for," concludes Guanco. "This highly functional NAS platform has exceeded my expectations in terms of supporting disk-based backup, recovery and disaster recovery. I'm looking forward to additional uses in the future."

Second – EMC Isilon doesn’t just break, but SHATTERS the world records for NFS and CIFS performance and scaling.

You can see the SPEC SFS results here, but here’s a summary table of some of the key results:

Vendor	Peak CIFS IOPS	CIFS ORT	Peak NFS IOPS	NFS ORT	Number of File Systems
EMC Isilon	1,612,778	2.34	1,112,705	2.54	1
EMC VNX	661,951	0.81	497,623	0.96	8
Huawei Symantec	712,664	1.81	636,036	0.99	8
IBM	--		403,326	3.23	1
NetApp	64,292	1.50	190,675	1.17	2

You can see that EMC is doing great here in two distinct categories.   VNX holds the record in the “swiss army knife” grouping (storage platforms that do it all) – which consists of few others, but certainly our respected colleagues at NetApp (FAS hardware running the ONTAP 7-mode software stack).

There’s the emerging category of large scale-out filesystem storage, where Isilon competes with some (IBM SONAS, and NetApp FAS hardware running the ONTAP cluster-mode software stack or NetApp Engenio hardware running things like Gluster), and competes very strongly.  

A little sidebar… NetApp – you are our primary NAS competitor, and a strong competitor for EMC in the “swiss army knife” category (VNX vs. 7-mode software stack on FAS hardware).  But clearly these massive scale-out global namespace models is one of the ways the filesystem world is evolving (in parallel to continued evolutions in the “swiss army knife” category).  Where are you?

Ok, back to the point… The key, however, about the Isilon result isn’t just that it’s huge (it is), or that it’s a single, global filesystem that can be PB in scale (it is), or that it’s 140 nodes in the cluster (which it is), or that it uses swank Isilon S200 hardware (Westmere, SSDs, SAS, 10GbE out the ying-yang).    

What’s a amazing to me is that:

1. the environment scaled linearly (and results were posted at 14, 28, 56 and 140 nodes scaling points)
2. customers can start small (and low cost) and easy.
3. and it stays easy as the cluster grows.   Filesystems spread out automatically across nodes as you add them.   Metadata gets distributed across nodes – including using SSDs preferentially where they help.   If you’ve never added a Isilon node – a trained monkey could do it.  Heck, forget the training – give it an IP, and go.  

Here’s a video that shows the cluster growing during the build for the SPEC SFS exercise.

Pretty darn cool if you ask me :-)

Comments

[TM Storage]

Hi chad, Check http://www.spec.org/sfs97r1/results/res2006q2/sfs97r1-20060522-00263.html

Welcome to the million IOPS club. 5 years later and leader who?
Make it 5 million and the tech world will accept you.
Why is the response time so bad for 100K to 600K where most of the competition can do it below 1.5ms?
SVC can do Million SAN IOPS 4 years ago as well.

Hope to get some real tech answer published.
- TM

[Richard]

Hello,

stop me if I'm wrong, but I think some important data are missing :
Disks numbers and network ports. Ok, the record is broken, but look at the hardware needed!
If you calculate the ratio between CIFS throughput and disks number or networks ports the results are a bit different!

Regards,

Richard

[Chad Sakac]

@Richard - thanks - I will go back and update the table. Would be interesting to see total cost, total power, etc. I wish there was a "complexity of config" (a huge parameter when it comes to scale).

@TM storage - I've seen that comment in a couple places, and it's LAUGHABLE. Let's do a little education.

First: sfs97r1 and SPEC SFS 2008 are TOTALLY DIFFERENT BENCHMARKS. It says right on spec: "SPECsfs2008 results may only be compared to other SPECsfs2008 results for the same protocol." Also, SPEC SFS 97 has been retired it's so old.

I'll make an analogy that "joe regular PC user" would understand - to compare a result from one to the other is like comparing 3Dmark06 and 3Dmark11 scores.

Second:latency on scale-out systems is inherently higher than it is in non-scale out systems (this is a system-level tradeoff).

As data is distributed across nodes, the internal operations that make up serving any given IO is also inherently distributed. Note how SONAS and Isilon have latencies that are 2x-3x higher than the "many filesystems" other leading entries - and also HAPPEN to be the only configs that show one massive scaled out filesystem.

THAT BTW, is one of the reasons that the "transactional swiss-army knife" systems like VNX and NetApp FAS running the ONTAP 7-mode stack exist, and will continue to exist. If you're an engineer with those "non-scale out stacks" you struggle with "how do I scale and manage at scale". If you're an engineer with the "scale-out stacks" you struggle with "how do I introduce more 'classic' functionality and improve small IO, transactional IO latency".

Third: SAN IOPs and SPEC SFS IOps are different. One is a NAS operation, one is a block IO SCSI operation. EMC transactional block subsystems have been able to do 1M+ IOps for eons. Comparing NAS with block is night and day. If you want the compare, put gluster on top of SVC and go for it.

Hope this helps, and highlights indeed why this is indeed a true landmark result (and why the comments are totally off base).

[TM Storage]

Hi Chad, thanks for explaining. The Spec97 results are retired and not discarded. Also the NFS versions are the major differences between old LADDIS nfs tests and sfs2008. Offcourse CiFS are new.
The results cannot be compared. Agreed! But 2006 vs 2011 things changes. Give credit and take credits.
But why do you compare your system with competitor in your article and mislead stating x% better when you know the number of disks and configs are insanely different.
It's like saying a flight is faster than a bicycle.
Got to compare say Boeing Vs Airbus to be fair. Compare your scaleout with other scale-out configs. Fair.

Also 3000 disks + SSD is a lot of disks. Guess you get my point.

[Christian BIrkle]

Hi Chad,

I would be nice if we could see specs on how well everage Joe configs of VNX system fare.

I work for an EMC distributor in Germany and we get asked questions about performance now and then.

I'd be the laughing stock if I'd show them these numbers from systems who hardly have a real work use case. Even higher ed customers or HPC users won't have that amount of money to go for such an Isilon system.

Regards
Christian

[udubplate]

Cool no doubt, but this sizing probably applies to a fraction of a percent of customers. Seems like manufacturers are competing to release benchmarks that don't apply to the majority of customer use cases. Wish there was an industry standard benchmark for a set sized real world system config so we could see some of the numbers that apply to the common folks as well as continue to see the cool "wow factor" results.

[Dimitris Krekoukias]

Hi all, D from NetApp here.

It is a funny comparison in the table, the CIFS result stated is from a low-end NetApp box with a few SATA disks + Flash Cache, that costs a very very small fraction of everything else in that table.

Chad, for completeness' sake, you should post number of disks, usable capacity as well as costs in the same table, so that people can make a more informed conclusion.

In general though, most people would not really buy scale-out NAS unless they want the throughput for large files (not IOPS) and sheer capacity. Most people that need something in the tens (or even hundreds) of TB will usually go for traditional NAS.

Thx

D

[Chad Sakac]

@Dimitris - the links to the SPEC SFS show all the details of the configs (number of disks etc). Usable capacity in Isilon configs tends to be very high. Likewise, it's not uncommon for an Isilon cluster to be able to drive 50+ GBps of bandwdith (your point of the primary use cases for scale-out NAS being bandwdith dominated, not IOps is also true - TODAY).

I actually agree that most customers go for "traditional" clustered NAS today (we certainly sell more VNX than we do Isilon). But, I would expect that over time, scaleout models will tend to be an expected architecture of NAS - at every scale. This seems inevitable as scale-out models gain some of the features they currently lack. After all - that seems to be the direction at Netapp too - if what I hear about ONTAP 8.1 is correct.

Personally, I'm surprised that NetApp hasn't posted any scale-out "cluster mode" results in ages. The scale out NAS space is booming.

[david owens]

Why does the article over at the Register - http://forums.theregister.co.uk/forum/1/2011/11/02/netapp_specsfs2008/

seem to contradict your post ?

From the Register ....

NetApp doesn't do scale-out clustering, everyone knows that. Oh but it does, and it has whacked Isilon firmly in its scaled-out cluster with a SPECSfs2008 benchmark that is 36 per cent faster using half the disks and 116 fewer nodes. EMC Isilon was the top SPECsfs2008 NFS dog with 1,112,705 IOPS recorded from a 140-node S200 …

I own one FC but am looking at both for CIFS

[Chad Sakac]

@David - thanks for the question. Warning - I don't claim to be a NetApp expert (they are welcome to comment on the thread)

The answer is twofold:

1) Timing - these benchmarks are always a game of leapfrog. For about a year, EMC had the top listing, and now we don't. You can fully expect another posting soon enough :-)

MORE IMPORTANTLY, HOWEVER:

2) NetApp still doesn't do scale-out NAS. They do federated namespace NAS. The difference is very material to some customers (but of course, that's up to the use case, and the customer).

I cover that difference here: http://virtualgeek.typepad.com/virtual_geek/2011/08/nfs-changes-in-vsphere-5-and-true-scale-out-nas-isilon.html.

The main points:

- Scale out-NAS can support a single filesystem, spread across all the nodes. Conversely, in a federated namespace (with "classic" clusters underneath), each NAS cluster has it's own RAID, it's own volumes (FlexVols in NetApp parlance), it's own filesystem. What they do is layer a global namespace on top.

The difference is that the performance for a given filesystem is limited by the node it lives on. THAT'S why you won't see many "single filesystem" postings from NetApp in cluster mode - because they way they get a high performance mark is to have many independent filesystem, linked into a global namespace. If you ran out of performance in a single filesystem - well, you're done. Conversely, what people EXPECT from scale-out models - you

- Scale-out NAS models mean you don't need to worry about the constructs of a node. No RAID, no volumes. You can destroy nodes and the performance impact is the "expected" model of "you go from 'N' to 'N-1'". Conversely, in a federated global namespace - the performance of the global namespace isn't linear, and if you lose a clustered pair in the underlying FAS cluster (i.e. a FAS2000, 3000, 6000 are all "pairs" of independent storage "brains", with hard attachments to the disks behind them in enclosures), boom, you lose that part of the fileystem. Isilon has each node containing the brains, cache, disk. As you add nodes, they all go up together. You can dial up the protection level to be able to withstand losing many nodes.

These are just SOME of the differences that are immediately obvious when you look under the covers. It's not to say that there aren't some advantages of NetApp's approach (for example, FOR NOW, lower single-IO latency) - but IMO (and warning, I'm biased here) - when you're REALLY talking about scale-out, simplicity and simplicity at petabyte scale depends on the underlying architectural things I'm pointing out. It's VERY difficult to "bolt that on" to legacy architectural models.

If you go back to that demo I did of vSphere 5 NFS behavior (link above in the comment) - imagine how that would work with the two VERY different models (scale-out NAS vs. federated namespace on top of clustered pairs)

Thanks for the question, and I hope that helps!