« A Multivendor Post to help our mutual NFS customers using VMware | Main | Why I work for EMC.. »

June 10, 2009


Feed You can follow this conversation by subscribing to the comment feed for this post.


Sorry to be nit-picky, but it would be more accurate to say that the FCoE *draft* standard is complete. The FC-BB-5 draft has been forwarded to INCITS for Public Review. The Public Review process is a 45-days period that leads to the adoption of the draft as an ANSI standard. So it's not quite soup yet.
Besides, FCoE was the easy part --- the underlying CEE is still a year out. The IEEE and IETF still need to finalize CEE, DCBX, and TRILL. The reality is that using Ethernet PAUSE or Per Priority Pause to recreate the flow control that's built in to Fibre Channel is going to be a lot of work from a standards perspective.

Chad Sakac

St0ragebear - that's not being nit-picky - that's VERY IMPORTANT. I originally pointed out the state of the lossless ethernet elements in the "read more" section, but on reflection after your comment - I decided to pull it up front more prominently.

It's up there - let me know if you think it's more clear.

Thanks again - and corrections to anything I write are ALWAYS welcome!

David Robertson

well we are in the process of building a new DC, and after EMC World, and getting a good idea about where FCoE is, I am strongly pushing that we adopt it for our vSphere infrastructure that will be primarily on blades, which i think is where FCoE will have a huge impact in more bandwidth to blades, with fewer cables.

Charlie Dellacona

Hi Chad,

Your post raised some questions for me.

What are the use cases that FCoE covers and iSCSI with 10Ge doesn't?

What does FCoE "bring to the table" that iSCSI with 10Ge doesn't?

TCP guarantees delivery, so how is iSCSI "lossy?"

In the FCoE/DCE stack is there *ever* a lost frame? Is there a re-transmit capability in that stack to handle lost or corrupt frames? That would sound "lossy" to me as well.

Is the sole advantage of FCoE quick recovery from transmit failures?


David Black

Hi Charlie,

I've worked on both the iSCSI and FCoE standards, so I'll try to answer your questions:

FCoE Use case: FC SANs are already installed, a new server rack arrives cabled only with 10Gig Ethernet, but the administrator doesn't want to configure or manage an iSCSI/FC gateway in order to access the existing storage systems.

What FCoE brings to the table: Transparent connectivity to existing FC SANs, including extension & reuse of existing FC-based management software and practices.

Of course, not every facility has FC SANs installed or wants to access existing FC-based storage from new servers. If one is starting from the proverbial "blank sheet of paper," the longer initial discussion should be about appropriateness of iSCSI and/or FC for the workload and facility (there's no single answer that covers all cases). For that sort of discussion, FCoE is a version of FC that runs over Ethernet. Management complexity and technology familiarity are considerations here (as they are in the use case above).

"lossy": Chad's wording about "lossy" is not the best ;-). What I think (hope) he meant to say is that by virtue of TCP, iSCSI copes much better with losses than FCoE. Losses are inevitable - sooner or later there will be a bad CRC (good links have very small bit error rates, but they're not zero).

Courtesy of TCP, iSCSI rides through an occasional loss nicely (e.g., TCP with SACK retransmits quickly without changing the window size or backing off). FCoE doesn't do anything at the FC level for disk I/O - eventually something at a higher level (e.g., a SCSI multipathing driver such as PowerPath) notices that the I/O didn't complete (i.e., times it out) and redrives the entire I/O. Before I get dinged for not mentioning it, tape (FC-TAPE) is different in that it can retransmit only a lost portion of an I/O at the FC level rather than having to redrive the entire SCSI I/O.

Because FCoE has no retransmission counterpart to TCP, it really has to run over CEE Ethernet that is designed and engineered (both links and network topology) to not drop packets for congestion or flow control reasons. iSCSI is much more tolerant of "stupid network tricks", and will run over "lossy" networks that one should never try to use FCoE on.

I hope this helps.

Charlie Dellacona


Thank you for your reply, it's nice to hear from an expert.

Re the use case, I was looking for something that FCoE could do that iSCSI could not that would be of value in the data center. I am not aware of any.

Overall your reply suggests that FCoE is an intermediary technology between FC and iSCSI to lessen disruption in transition. If that's the case, I'd have to wonder whether it was worth the effort that network and HBA vendors put into it and the expense customers will be hit with to adopt it. Intermediate steps to the future just bring extra cost and delay the benefit of adoption; particularly so when there is no re-use in the intermediate technology. Usually its better to just bite the bullet.

David Black


Since FCoE (via FCP) and iSCSI are both SCSI transports, it's not surprising that they have similar functionality. Since you asked, one thing that FCoE can do in principle that iSCSI cannot do is mainframe storage (i.e., FICON over FCP). Whether that'll happen is up to the mainframe folks, but it is a distinct possibility.

There are a variety of views on the relative roles of FC and iSCSI; yours seem to run towards the "iSCSI will make FC obsolete" end of the spectrum. I believe that the two technologies will co-exist for the foreseeable future (3-5 years). When FC is the storage networking technology of choice, FCoE offers significant benefits for rack-scale server integration, which is a clear trend.

Matthew Reed

Chad, I have question for you and since you have experience with iSCSI, what is your opinion concerning isolating iSCSI traffic? The reason I am asking is we recently encountered a serious latency issue with one of our ESX clusters that's connected to a Dell MD3000i JBOD. The interconnects are (2) CAT3750G switches that are stacked serving other Windows clients along with the ESX hardware initiators and the MD3000i targets, though they are in a dedicated VLAN. However, the VLAN tagging I found was being processed at our core (CAT6509)which is located across the street and the uplinks traverse approx 900 feet of dark fiber.

Now the problems started to really manifest just shortly after introducing the second ESX node because initially we thought our performance issues were due to a lack of server compute resources in trying to serve 150 VMs on one host. I know that sounds like a lot but a majority of those VMs are delta images and Linux virtual routers from our VM Lab Manager environment. However, the performance worsened (the ESXTOP command queues were 10-20K ms) even while VMs were turned off, so we disconnected the ESX hosts from the switches and went direct to the MD3000i and the difference was immediate as the command queues dropped to an avg of 4-10ms.

Now we think the combination of the VLAN overhead and the fact the gateway was in another building was no doubt attributing to this. However, our network guy as usual doesn't buy it and doesn't think we need to have a parallel dedicated network.

I would really like your opinion and or what your recommendation would be in our case. Also, we engaged Dell on their PowerConnect switches as they are cheap alternative to Cisco.

Thank you

Chad Sakac

@Matthew - uggh. Whenever I say "make sure your switches have enough port buffers"... What I am kinda saying is "be careful if using 3750s" (the Catalyst 6500s are much, much beefier).

Don't underestimate the performance needed by the lab manager use case - although they are deltas (Lab Manager uses the linked clone functionality hidden in ESX for some time, and now used by VMware View Composer as well), it doesn't change the IO (MBps/IOps) they need, only the capacity they need.

The distance thing isn't necessarily the issue - I would be surprised. Latency of a link (particularly of dark fiber connection) is microseconds. So long as it's not routed, it's not deadly.

BUT the model where you can't have good control of the network is very tough in the "when doing iSCSI/NFS for VMware - follow 'bet the business ethernet' infrastructure" guidelines.

The long and short of it - when iSCSI moves from "playing with it" to "building my storage network", you should build a IP storage network that looks, topologically, a lot like an FC network.

I would either use dedicated switches or basic port-based VLANs. Sounds to me (more information is needed) like you've got a private VLAN.

It's funny, and I am a SUPER iSCSI fan, but many, many times, the initial "oh, if we go iSCSI over FC, we could save $500 per port!" benefit quickly evaporates when it gets hairy.

Personally, if it were me, I would bring the data to the networking guy, and highlight that taking the network out of the loop resolved the issue. If he didn't want to buy it (this will be an interesting thing as the Ethernet and FC networks converge), then I would say "screw it", and just build an isolated iSCSI network with dedicated switches.

Dedicated Server

It's been almost 18 months, Have there been any glaring vulnerabilities with FCoE? I'm about to do a major upgrade and wanted to see if there were any huge problems.

Chad Sakac

@Dedicated Server - I think you're spam, but not sure :-)

I'll give the benefit of the doubt.

I know of many customers who are happily now deploying FCoE. Many of the early bumps are out.

The comments to this entry are closed.

  • BlogWithIntegrity.com


  • The opinions expressed here are my personal opinions. Content published here is not read or approved in advance by Dell Technologies and does not necessarily reflect the views and opinions of Dell Technologies or any part of Dell Technologies. This is my blog, it is not an Dell Technologies blog.