« June EMC/VMware Webcastssolid | Main | 3 Pieces of Bad News and a Deep Dive into APD »

June 27, 2011


Feed You can follow this conversation by subscribing to the comment feed for this post.

TM Storage

Hi chad, Check http://www.spec.org/sfs97r1/results/res2006q2/sfs97r1-20060522-00263.html

Welcome to the million IOPS club. 5 years later and leader who?
Make it 5 million and the tech world will accept you.
Why is the response time so bad for 100K to 600K where most of the competition can do it below 1.5ms?
SVC can do Million SAN IOPS 4 years ago as well.

Hope to get some real tech answer published.
- TM



stop me if I'm wrong, but I think some important data are missing :
Disks numbers and network ports. Ok, the record is broken, but look at the hardware needed!
If you calculate the ratio between CIFS throughput and disks number or networks ports the results are a bit different!



Chad Sakac

@Richard - thanks - I will go back and update the table. Would be interesting to see total cost, total power, etc. I wish there was a "complexity of config" (a huge parameter when it comes to scale).

@TM storage - I've seen that comment in a couple places, and it's LAUGHABLE. Let's do a little education.

First: sfs97r1 and SPEC SFS 2008 are TOTALLY DIFFERENT BENCHMARKS. It says right on spec: "SPECsfs2008 results may only be compared to other SPECsfs2008 results for the same protocol." Also, SPEC SFS 97 has been retired it's so old.

I'll make an analogy that "joe regular PC user" would understand - to compare a result from one to the other is like comparing 3Dmark06 and 3Dmark11 scores.

Second:latency on scale-out systems is inherently higher than it is in non-scale out systems (this is a system-level tradeoff).

As data is distributed across nodes, the internal operations that make up serving any given IO is also inherently distributed. Note how SONAS and Isilon have latencies that are 2x-3x higher than the "many filesystems" other leading entries - and also HAPPEN to be the only configs that show one massive scaled out filesystem.

THAT BTW, is one of the reasons that the "transactional swiss-army knife" systems like VNX and NetApp FAS running the ONTAP 7-mode stack exist, and will continue to exist. If you're an engineer with those "non-scale out stacks" you struggle with "how do I scale and manage at scale". If you're an engineer with the "scale-out stacks" you struggle with "how do I introduce more 'classic' functionality and improve small IO, transactional IO latency".

Third: SAN IOPs and SPEC SFS IOps are different. One is a NAS operation, one is a block IO SCSI operation. EMC transactional block subsystems have been able to do 1M+ IOps for eons. Comparing NAS with block is night and day. If you want the compare, put gluster on top of SVC and go for it.

Hope this helps, and highlights indeed why this is indeed a true landmark result (and why the comments are totally off base).

TM Storage

Hi Chad, thanks for explaining. The Spec97 results are retired and not discarded. Also the NFS versions are the major differences between old LADDIS nfs tests and sfs2008. Offcourse CiFS are new.
The results cannot be compared. Agreed! But 2006 vs 2011 things changes. Give credit and take credits.
But why do you compare your system with competitor in your article and mislead stating x% better when you know the number of disks and configs are insanely different.
It's like saying a flight is faster than a bicycle.
Got to compare say Boeing Vs Airbus to be fair. Compare your scaleout with other scale-out configs. Fair.

Also 3000 disks + SSD is a lot of disks. Guess you get my point.

Christian BIrkle

Hi Chad,

I would be nice if we could see specs on how well everage Joe configs of VNX system fare.

I work for an EMC distributor in Germany and we get asked questions about performance now and then.

I'd be the laughing stock if I'd show them these numbers from systems who hardly have a real work use case. Even higher ed customers or HPC users won't have that amount of money to go for such an Isilon system.



Cool no doubt, but this sizing probably applies to a fraction of a percent of customers. Seems like manufacturers are competing to release benchmarks that don't apply to the majority of customer use cases. Wish there was an industry standard benchmark for a set sized real world system config so we could see some of the numbers that apply to the common folks as well as continue to see the cool "wow factor" results.

Dimitris Krekoukias

Hi all, D from NetApp here.

It is a funny comparison in the table, the CIFS result stated is from a low-end NetApp box with a few SATA disks + Flash Cache, that costs a very very small fraction of everything else in that table.

Chad, for completeness' sake, you should post number of disks, usable capacity as well as costs in the same table, so that people can make a more informed conclusion.

In general though, most people would not really buy scale-out NAS unless they want the throughput for large files (not IOPS) and sheer capacity. Most people that need something in the tens (or even hundreds) of TB will usually go for traditional NAS.



Chad Sakac

@Dimitris - the links to the SPEC SFS show all the details of the configs (number of disks etc). Usable capacity in Isilon configs tends to be very high. Likewise, it's not uncommon for an Isilon cluster to be able to drive 50+ GBps of bandwdith (your point of the primary use cases for scale-out NAS being bandwdith dominated, not IOps is also true - TODAY).

I actually agree that most customers go for "traditional" clustered NAS today (we certainly sell more VNX than we do Isilon). But, I would expect that over time, scaleout models will tend to be an expected architecture of NAS - at every scale. This seems inevitable as scale-out models gain some of the features they currently lack. After all - that seems to be the direction at Netapp too - if what I hear about ONTAP 8.1 is correct.

Personally, I'm surprised that NetApp hasn't posted any scale-out "cluster mode" results in ages. The scale out NAS space is booming.

david owens

Why does the article over at the Register - http://forums.theregister.co.uk/forum/1/2011/11/02/netapp_specsfs2008/

seem to contradict your post ?

From the Register ....

NetApp doesn't do scale-out clustering, everyone knows that. Oh but it does, and it has whacked Isilon firmly in its scaled-out cluster with a SPECSfs2008 benchmark that is 36 per cent faster using half the disks and 116 fewer nodes. EMC Isilon was the top SPECsfs2008 NFS dog with 1,112,705 IOPS recorded from a 140-node S200 …

I own one FC but am looking at both for CIFS

Chad Sakac

@David - thanks for the question. Warning - I don't claim to be a NetApp expert (they are welcome to comment on the thread)

The answer is twofold:

1) Timing - these benchmarks are always a game of leapfrog. For about a year, EMC had the top listing, and now we don't. You can fully expect another posting soon enough :-)


2) NetApp still doesn't do scale-out NAS. They do federated namespace NAS. The difference is very material to some customers (but of course, that's up to the use case, and the customer).

I cover that difference here: http://virtualgeek.typepad.com/virtual_geek/2011/08/nfs-changes-in-vsphere-5-and-true-scale-out-nas-isilon.html.

The main points:

- Scale out-NAS can support a single filesystem, spread across all the nodes. Conversely, in a federated namespace (with "classic" clusters underneath), each NAS cluster has it's own RAID, it's own volumes (FlexVols in NetApp parlance), it's own filesystem. What they do is layer a global namespace on top.

The difference is that the performance for a given filesystem is limited by the node it lives on. THAT'S why you won't see many "single filesystem" postings from NetApp in cluster mode - because they way they get a high performance mark is to have many independent filesystem, linked into a global namespace. If you ran out of performance in a single filesystem - well, you're done. Conversely, what people EXPECT from scale-out models - you

- Scale-out NAS models mean you don't need to worry about the constructs of a node. No RAID, no volumes. You can destroy nodes and the performance impact is the "expected" model of "you go from 'N' to 'N-1'". Conversely, in a federated global namespace - the performance of the global namespace isn't linear, and if you lose a clustered pair in the underlying FAS cluster (i.e. a FAS2000, 3000, 6000 are all "pairs" of independent storage "brains", with hard attachments to the disks behind them in enclosures), boom, you lose that part of the fileystem. Isilon has each node containing the brains, cache, disk. As you add nodes, they all go up together. You can dial up the protection level to be able to withstand losing many nodes.

These are just SOME of the differences that are immediately obvious when you look under the covers. It's not to say that there aren't some advantages of NetApp's approach (for example, FOR NOW, lower single-IO latency) - but IMO (and warning, I'm biased here) - when you're REALLY talking about scale-out, simplicity and simplicity at petabyte scale depends on the underlying architectural things I'm pointing out. It's VERY difficult to "bolt that on" to legacy architectural models.

If you go back to that demo I did of vSphere 5 NFS behavior (link above in the comment) - imagine how that would work with the two VERY different models (scale-out NAS vs. federated namespace on top of clustered pairs)

Thanks for the question, and I hope that helps!

The comments to this entry are closed.

  • BlogWithIntegrity.com


  • The opinions expressed here are my personal opinions. Content published here is not read or approved in advance by Dell Technologies and does not necessarily reflect the views and opinions of Dell Technologies or any part of Dell Technologies. This is my blog, it is not an Dell Technologies blog.