Scott Lowe recently wrote a good post on FCoE, and his thoughts here. The comments of his readers are comments I’ve heard from others as well, so I posted a response in the comments, but I think Scott and I don’t have the same readership (and perhaps those that do may not read the comments)
This is an important dialog, IMHO, and I thought the my response was worth posting, as I’ve gotten loads of questions like this also.
If you’re interested in this thread – suggest reading Scott’s posts and the comments. If you want to see my take, read on….
(from my comment on Scott’s blog post)
Guys The “multi-hop” thing is a bit old news - I did a big post on this when the FCoE spec was done (June 3rd)
This covers it in gory detail.
The specific issue is that “pre standard” initiators and targets were missing something called FIP (FCoE initialization protocol). The gen 1 HBAs from Qlogic and Emulex were really more for early interop, plugfests, and development, and I believe (I know this for a fact for the Qlogic 8000 series - and I would fully expect the same from Emulex) are not software upgradable to the FC-BB-5 standard that includes FIP.
BTW - we caught flack at EMC for not natively supporting FCoE earlier on the array targets, but this was why - the standard simply wasn’t ready. It was ready for host-FCoE switch-FC switch-FC target. Now, it’s getting ready for array targets. Personally, that’s why I disagreed with the approach of taking the QLE8000 series card (with custom pre-FIP standard elements), putting into a FAS head, and calling that a solution. While that was going on (and making marketing noise - but frankly a move that doesn’t help the customer, because now they have a FAS head that needs a heavy hardware maintenance window to do a PCIe card upgrade), we were busy doing interop and working on the standard at the standard body (look at the meeting minutes - they are all public).
We’re now, of course, developing an ultraflex IO module for FCoE, which are hot-swappable.
But back to the larger question - why FCoE? People who know me, I’m a SUPER fan of NAS and iSCSI, and naturally am biased in that direction, but as I’ve worked with more and more customers, I have a growing understanding of the why.
NFS is great and iSCSI are great, but there’s no getting away from the fact that they depend on TCP retransmission mechanics (and in the case of NFS, potentially even higher in the protocol stack if you use it over UDP - though this is not supported in VMware environments today). because of the intrinsic model of the protocol stack, the higher you go, the longer the latencies in various operations. One example (and it’s only one) - this means always seconds, and normally many tens of seconds for state/loss of connection (assuming that the target fails over instantly, which is not the case of most NAS devices). Doing it in shorter timeframes would be BAD, as in this case the target is an IP, and for an IP address to be non-reachable for seconds - is NORMAL.
There’s also the question of the fact that anything dependent on TCP/IP also will have scenarioes that depend on ARPs, which can take time.
This isn’t a secret. Look at the Netapp TR-3428 (and upcoming TR-3749) and EMC H6337 docs which spell the timeouts for NFS datastores on FAS and Celerra platforms respectively - which are in many tens of seconds (refer to the latest – currently it adds up to 125 seconds), and for iSCSI – if you read the VMware guides, the recommendation is 60 seconds.
FCoE expects most transmission loss handling to be done at the Ethernet layer, via 802.1Qbb (STILL NOT A STANDARD!) for lossless congestion handling and legacy CRC mechanisms for line errors. This means milliseconds - and in fact in many cases microseconds of link state sensitivity.
Also, whereas we are seeing 30x performance increases for solid state disk on devices without filesystems, we see 4-6x in cases where they support a filesystem. That doesn’t mean filesystems (or NAS devices are bad), but highlights that one answer isn’t the answer all the time, for all workloads, all SLAs, and all use cases.
These ARE NOT showstoppers for many, many (most?) applications, and many, many use cases, but they are for some - and often, those are for applications with hyper-stringent SLAs - but we want to virtualize everything, ever application possible, right?
All FCoE adapters and switches can also be used for iSCSI and NAS, so don’t think of it as an either/or, but an “and”. It means that it is possible to whittle the use cases that can’t use an ethernet storage transport to near zero (it’s not zero, because there will always be mainframes and whatnot). The ultimate point on this (”this” being the point that it’s not an FC “HBA”, but rather a “NIC feature”) is that Intel has commited to supporting the final result of 802.1Qbb and then doing a software initiator - at that point, FCoE support will just be an attribute of every commodity NIC and switch on the market. Everyone in the FC HBA/switch market is rushing to it not because they want proprietary, but rather because were reaching the inflection point where if you’re not doing this, you’re going to be out of business (maybe not today, but at a relatively near “tomorrow”).
The FCoE idea important (again as a “NIC/switch feature”, because it means that convergence (wire once, use for LAN/NAS/iSCSI/FCoE) is then applicable to a broader market, which only accelerates the broader use of ethernet storage, which many people (me included) want to see come sooner rather than later.
There’s a far lesser IT value proposition of maintaining and integrating with exisitng tools and processes. I only say lesser because frankly, if there’s a better way, it can over time change a process.
Remember - this is coming from someone who:
a) loves NAS
b) loves iSCSI (came from iSCSI startup)
c) works for a storage company that is in the NAS, iSCSI, FC, and FCoE (and heck, COS and CAS as well) business - we just do what our customers tell us they need.
At least in my personal experience, our customers are asking for FCoE for those reasons.