Why FCoE? Why not just NAS and iSCSI?

Scott Lowe recently wrote a good post on FCoE, and his thoughts here.   The comments of his readers are comments I’ve heard from others as well, so I posted a response in the comments, but I think Scott and I don’t have the same readership (and perhaps those that do may not read the comments)

This is an important dialog, IMHO, and I thought the my response was worth posting, as I’ve gotten loads of questions like this also.

If you’re interested in this thread – suggest reading Scott’s posts and the comments.  If you want to see my take, read on….

(from my comment on Scott’s blog post)

Guys The “multi-hop” thing is a bit old news - I did a big post on this when the FCoE spec was done (June 3rd)

http://virtualgeek.typepad.com/virtual_geek/2009/06/fcoe-ratified.html

This covers it in gory detail.

The specific issue is that “pre standard” initiators and targets were missing something called FIP (FCoE initialization protocol). The gen 1 HBAs from Qlogic and Emulex were really more for early interop, plugfests, and development, and I believe (I know this for a fact for the Qlogic 8000 series - and I would fully expect the same from Emulex) are not software upgradable to the FC-BB-5 standard that includes FIP.

BTW - we caught flack at EMC for not natively supporting FCoE earlier on the array targets, but this was why - the standard simply wasn’t ready. It was ready for host-FCoE switch-FC switch-FC target. Now, it’s getting ready for array targets. Personally, that’s why I disagreed with the approach of taking the QLE8000 series card (with custom pre-FIP standard elements), putting into a FAS head, and calling that a solution. While that was going on (and making marketing noise - but frankly a move that doesn’t help the customer, because now they have a FAS head that needs a heavy hardware maintenance window to do a PCIe card upgrade), we were busy doing interop and working on the standard at the standard body (look at the meeting minutes - they are all public).

We’re now, of course, developing an ultraflex IO module for FCoE, which are hot-swappable.

But back to the larger question - why FCoE? People who know me, I’m a SUPER fan of NAS and iSCSI, and naturally am biased in that direction, but as I’ve worked with more and more customers, I have a growing understanding of the why.

NFS is great and iSCSI are great, but there’s no getting away from the fact that they depend on TCP retransmission mechanics (and in the case of NFS, potentially even higher in the protocol stack if you use it over UDP - though this is not supported in VMware environments today). because of the intrinsic model of the protocol stack, the higher you go, the longer the latencies in various operations. One example (and it’s only one) - this means always seconds, and normally many tens of seconds for state/loss of connection (assuming that the target fails over instantly, which is not the case of most NAS devices). Doing it in shorter timeframes would be BAD, as in this case the target is an IP, and for an IP address to be non-reachable for seconds - is NORMAL.

There’s also the question of the fact that anything dependent on TCP/IP also will have scenarioes that depend on ARPs, which can take time.

This isn’t a secret. Look at the Netapp TR-3428 (and upcoming TR-3749) and EMC H6337 docs which spell the timeouts for NFS datastores on FAS and Celerra platforms respectively - which are in many tens of seconds (refer to the latest – currently it adds up to 125 seconds), and for iSCSI – if you read the VMware guides, the recommendation is 60 seconds.

FCoE expects most transmission loss handling to be done at the Ethernet layer, via 802.1Qbb (STILL NOT A STANDARD!) for lossless congestion handling and legacy CRC mechanisms for line errors. This means milliseconds - and in fact in many cases microseconds of link state sensitivity.

Also, whereas we are seeing 30x performance increases for solid state disk on devices without filesystems, we see 4-6x in cases where they support a filesystem. That doesn’t mean filesystems (or NAS devices are bad), but highlights that one answer isn’t the answer all the time, for all workloads, all SLAs, and all use cases.

These ARE NOT showstoppers for many, many (most?) applications, and many, many use cases, but they are for some - and often, those are for applications with hyper-stringent SLAs - but we want to virtualize everything, ever application possible, right?

All FCoE adapters and switches can also be used for iSCSI and NAS, so don’t think of it as an either/or, but an “and”. It means that it is possible to whittle the use cases that can’t use an ethernet storage transport to near zero (it’s not zero, because there will always be mainframes and whatnot). The ultimate point on this (”this” being the point that it’s not an FC “HBA”, but rather a “NIC feature”) is that Intel has commited to supporting the final result of 802.1Qbb and then doing a software initiator - at that point, FCoE support will just be an attribute of every commodity NIC and switch on the market. Everyone in the FC HBA/switch market is rushing to it not because they want proprietary, but rather because were reaching the inflection point where if you’re not doing this, you’re going to be out of business (maybe not today, but at a relatively near “tomorrow”).

The FCoE idea important (again as a “NIC/switch feature”, because it means that convergence (wire once, use for LAN/NAS/iSCSI/FCoE) is then applicable to a broader market, which only accelerates the broader use of ethernet storage, which many people (me included) want to see come sooner rather than later.

There’s a far lesser IT value proposition of maintaining and integrating with exisitng tools and processes. I only say lesser because frankly, if there’s a better way, it can over time change a process.

Remember - this is coming from someone who:

a) loves NAS
b) loves iSCSI (came from iSCSI startup)
c) works for a storage company that is in the NAS, iSCSI, FC, and FCoE (and heck, COS and CAS as well) business - we just do what our customers tell us they need.

At least in my personal experience, our customers are asking for FCoE for those reasons.

Comments

[T]

And don't forget that interop testing is also underway for using DCB with iSCSI and NAS as well.

[Chad Sakac]

Absolutely T, and I would expect some improvements in all use cases from DCB (this is Datacenter Bridging for those who didn't read the thread or the original posts - it's the IEEE word for what is called Datacenter Ethernet/DCE or Converged Enhanced Ethernet/CEE).

That said, as soon as you are using the IP stack for core functions (like iSCSI does for iSCSI initiator/target network portals), some of the behavior is intrinsic (again, not good/bad, just different), and the NAS code paths aren't the same as the block code paths in the devices that are out there (again, not good/bad, just different) that expresses itself as "different behavioral envelopes" (or put other wise certain things are harder on NAS than block, and certain things are harder on block than NAS.

[Simon Rohrich]

The new Cisco UCS will drive the FCoE adoption. It will also make our model of data center build outs using micro containers work very well.

[David Robertson]

I have been pushing our network guys to go all Nexus and FCoE in our new datacenter, and I think I have them on my side, will be exciting times shortly for us.

[Chad Sakac]

David - cool, definitely please post your experiences. I just bought 6 N5Ks, almost 200 FCoE CNAs, and a couple UCS chassis, will be sharing the experiences also!

[Brad Hedlund]

Chad,
The reliance on TCP can take on an order-of-magnitude increase in complexity and processing requirements at 10GE vs. 1GE. If you understand how TCP windowing affects throughout you can calculate that in order to maintain 10GE throughput at 20-30msec latencies requires as much as 32MB buffers on the TOE - driving up adapter costs. FCoE does not have this issue. Perhaps next SSD arrays with much lower latencies will provide some relief for iSCSI @ 10GE. Until then, FCoE has the clear advantage @ 10GE, IMHO.

Cheers,
Brad

[Charlie Dellacona]

Chad,

Thanks for continuing this conversation. NFS is a bigger animal, it does things that iSCSI, FC or FCOE do not do. The real question is simply why FCOE with all its needs when there is already proven iSCSI and infrastructure?

ARPs are not an issue, once resolved they stay in caches for weeks. The timeouts are settable and the values can be hardwired in the tables if necessary. Red herring.

You keep eluding to customers who have use cases where iSCSI is not acceptable. Please tell us about one with hyper stringent SLAs in detail. While you're doing that, make sure its doesn't have IP in its critical path anywhere else (like DNS or AD or between the App tiers or the front end connection to actual users). If TCP/IP is acceptable on the front end then it is on the backend as well, a timeout is a timeout.

TCP with SACK eliminates a lot of the problems you are talking about.

Including FCoE in every NIC and switch just needlessly adds cost, complexity, and the bugs they bring when iSCSI already exists and uses proven commodity infrastructure.

As for wiring once, I can already do that with normal NICs and switches because I don't need FC compatability with iSCSI.

You may care to promote ethernet as a medium, but that's not the big value tock, it's TCP/IP as a protocol. That is what turns storage from an island into a network and will bring customers actual value.

"Ships in the night" protocols over a special case single media is NOT convergence. Solutions that leverage commodity hardware, run anywhere in the infrastructure, eliminate specialist skill sets, and aggregate technology demand to drive down cost, that's convergence.

--Charlie

p.s. Sorry I took so long to reply.

p.p.s. I realize the FCoE thing is going to happen. But long ago I railed against ATM for the same reasons. For a while that won too, then it died. I think in part because people kept pointing out it wasn't necessary.

[Evan Unrue]

Correct me if I'm mistaken here, as I'm still digging through FCoE and learning. My understanding is that currently if I have a datacentre with a significant investment in an FC SAN fabric (Cisco MDS for example), existing FC SAN skillsets inhouse, and a fair few racks of infrastructure; FCoE will allow me to retain my existing investment in my big MDS9500's and use FCoE to compliment my existing FC Fabric, but consolidate the fabric at the edge using FCoE for less complexity at the access layer. Also, my SAN admins still get to manage the SAN in a similar fashion as they normally would.

This sits well, as I would imagine that Data Centers are more likely to want to compliment an existing investment they have in FC infrastructure, while simplifying things at the edge; rather than do a complete tech refresh.

My question relates to complete greenfield sites, not just limited to the top end datacenter. Bearing in mind that using 10GbE we have significantly more bandwidth at the edge, the overhead involved with TCP/IP may be more than acceptable for some, it comes back to ensuring storage is provisioned over a reliable medium with zero or near zero packet drops. If 802.1qbb is a network function then surely the same 'no drop' CoS value PFC uses can be applied to iSCSI traffic rather than just FCoE? (that's an assumption, i couldn't find a definitive answer), which kind of makes the whole loss-less Ethernet function, less relevant to FCoE as a differentiator ? and iSCSI over 10GbE also a potentially viable storage protocol to facilitate all those great things that the IO consolidation brings ?

With that in mind, what other arguements are there for FCoE as opposed to iSCSI? I gather Cisco will be doing some pretty cool stuff with it with the UCS servers and most likely VMWare. Looking for an answer without vendor bias is proving tricky.

I work for an organisation which will be no doubt pushing the whole FCoE message soon enough. This is an objection I may well have to handle.. I'm keen to understand further

[Kurt]

Evan,
This article below answers a lot of the questions you have. Keep in mind that Cisco has created the article and pushes their data center design, and it seems more appropriate to existing FC installations. Over time, greenfields will surely be TCP/IP based, I believe, and concur with Charlie that this FCoE accomodation is exactly like the old ATM argument. It is just going to take some time. Check out Brocade's website for confirmation that they know full well that native FC is in decline.

http://www.cisco.com/en/US/docs/ECATS/Solutions_Guides/White_Papers/FCoE_NetApp_V12.pdf

[TriGuy]

So I wonder if someone could get Nexus like performance and infact even better performance at almost 1/2 the cost, while supporting Layer 3 at the ToR, EoR, or Middle of Row, and capabilities to vitualize this switch with another switch up to 70km apart would they do it. And in regards to the tech refesh for existing FC_SAN customers - hello you already have two or three 802.3 NICs in those same servers anyway connected to CORE LAN switch. One could easily upgrade their LAN switch to 10Gbp and the NICs for just the cost of the support on the FC_SAN director. The Net Net, is that alot of customers see the FCoE as an also ran to iSCSI. As such why wait? There are scalable 10Gb switches that can be upgraded to 40Gb and 100Gb. Combined with advanced QoS and LAN Switch virtualization technologies, one can now connect these moster LAN switches without Spanning Tree, HSRP, VRRP, between data centers up to 70km. Wow imagine that, vMotion and an iSCSI target being swapped over standard Ethernet between hot hot data centers at 70 clicks.