Virtual Geek

FCoE – Ratified!

UPDATED June 10th, 2009 to incorporate more on lossless Ethernet (was in “read more” section, pulled to front)

Congratulations to everyone who worked on the standard.

The FCoE standard celebrates its one week anniversary today - the T11 standards body ratified FCoE as a standard in FC-BB-5 on Wednesday June 3rd.

I’m not so much an FCoE true believer so much as I am an Ethernet true believer.   Coming to EMC from an iSCSI startup – I guess it was in the watercooler :-)   As I’ve interacted with more customers, I’ve gained a better understanding on why FCoE is important.  It offers a chance for a unified interconnect – covering a gamut of use cases.  Since all FCoE adapters are by definition Converged Network adapters – customer can use one, and use it for 10GbE LAN, 10GbE NAS, 10GbE iSCSI, and Fibre Channel via FCoE.

I still stick by the bet I made with Chuck (come on iSCSI!) – he and I disagree often, but boy the conversation is always be fun :-)

http://chucksblog.typepad.com/chucks_blog/2007/10/yes-we-occasion.html

And you can see I’ve been interested in this for some time:

http://virtualgeek.typepad.com/virtual_geek/2008/06/10-gigabit-ethe.html

(BTW – this was June 2008 - was after a couple of months of being “deep dive” introed to what was at that point the very confidential Cisco project codenamed California)

So why is it important?   On it’s own – iSCSI does not offer what FCoE brings to the table – a real chance for all current use cases/workloads - to consolidate the networks (by eliminating the “but what about this host here”… excuse) which in turn enables reductions in cable and port count, lower space/power requirements, and enable getting to a “cable once” (at least within major generational changes) model.  

This doesn’t mean I’m saying iSCSI is bad or even “not as good” as FCoE.   iSCSI is undoubtedly less expensive, and runs on a much, much broader set of equipment.  iSCSI is the fastest growing storage segment by a long shot for all these reasons, and is often the protocol of choice for customers with no existing shared storage infrastructure.   EMC is (and certainly I count myself in that) a huge iSCSI supporter.   However, iSCSI does fundamentally have lossy characteristics (using TCP retransmits for data integrity), longer TCP/IP timeout characteristics and other   But, without being able to cover the remaining use cases for FC – there wouldn’t be an opportunity for convergence of transport.   FCoE is that opportunity.

UPDATE:

Now, there’s still more work to be done – as St0ragebear pointed out in the comments, and I think it’s worth pulling up to the front part of this article.  First of all – now that the T11 standard is complete, there are other steps for it to become an ANSI standard – including a public review period.    The bigger point I had originally in the “read more” section, but I will pull up here.   Lossless Ethernet (CEE, DCE, IEEE Datacenter Bridging) is still not a standard, and this will take more time.   This is an important part of the FCoE idea.  The IEEE 802.1Qbb (Priority Flow Control) project is approved, but the standard is not done.  You can find more about that here: http://www.ieee802.org/1/pages/dcbridges.html … Now back to the original text….

I’m a glutton for punishment and love this stuff – it’s fascinating to look back at the minutes, see the progress, who’s driving what, seeing company names change (Nuova Systems).  You can see that all here: http://www.t11.org/t11/docreg.nsf/gfcoemi?OpenView.   It’s an easy way to see who’s driving, who’s participating (and how actively) and who’s following – at least from an engineering and standards standpoint.

 

Stuart Miniman from our office of the CTO (who was part of the standard process) does a very good video update on FCoE (including examples of Qlogic Gen 1 and Gen 2 Converged Network Adapters or CNAs):

What does this have to do with VMware?   VMware is one of the earliest, most potent use cases for FCoE and 10GbE generally.   I said it back in June 2008 (and have said as much at various VMworld sessions with Cisco and VMware) and I’ll say it again – massive consolidation, coupled with massive multicore and huge cheap RAM moves bottlenecks to the server I/O layer.   It’s fun to look back a year later and be borne out by what’s happened since.

EMC has been supporting FCoE for some time, and now with the standard ratified, it’s exciting times!  

Read on for more gory details!

A “Multivendor Post” to help our mutual NFS customers using VMware

UPDATED – June 11th, 8:21pm – new Celerra vSphere Best Practices posted

We were quite a bit surprised to see how popular our “Multivendor iSCSI” post was. The feedback was overwhelming and very supportive of industry leaders partnering to ensure customer’s success with VMware. While writing that post, we (Vaughn Stewart from NetApp and Chad Sakac from EMC) discussed following up the iSCSI post with one focused on deploying VMware over NFS. The most difficult part around creating this post is that we couldn’t do it with our iSCSI-focused colleagues.

Since the original post, we’ve been busy assisting our customers and partners. We apologize for the delay, so without further ado we present to you the followup: a “Multivendor NFS” post for our joint customers. One of the goals of this post is to dispel the FUD customers often hear around NFS. Heck, if EMC and NetApp can agree – then you KNOW this post is FUD-Free!

We would like to thank Stu Baker and Satyam Vaghani from VMware, along with numerous folks at EMC and NetApp for their input on this post.

While any NFSv3 server will work with VMware, and there are many NFS servers on the ESX HCL, there is a significant difference between what one can do with an enterprise class NFS storage array from EMC or NetApp. The reality is only NetApp and EMC are supporting NFS deployments with VMware in significant volume.

Both of us personally are big supporters of NFS for VMware – but if you look at our post histories - we’re both also rational and try our best (we’re human, so sometimes we fail) to be balanced and neutral (in my case, this is a balance to my VMFS post here).   We try to be good pragmatic voices, so our goal here is pragmatism and facts to help our mutual customers.

For more – read on…

vSphere and 2TB LUNs – changes from VI3.x

Literally last night, I was working furiously on the chapter for the book between my day job, and noted some really weird behavior in vSphere 4 with 2TB LUNs that I didn’t expect.  I poked at it, filed it away for further investigation.

Then today, a customer in france reported a case where a large MetaLUN was having an issue being partition – and stranger still, when they presented the LUN to an ESX 3.5 host, it worked fine, and then they could bring it back to the vSphere host.   I thought they might be related.   Sure enough they are.

I pinged my fellow EMCers and VMware folks – and there also a thread on this on VMTN here: http://communities.vmware.com/message/1269655#1269655

Clearly – this is something people are going to run into – if we all ran into it in the last couple of days.

So what the scoop with vSphere behavior at, slightly under, and over the 2TB LUN boundary case?

It’s no longer accurate to say that the largest single partition in vSphere is 2TB (this is a function of the LVM which uses a CHS partitioning scheme – not of VMFS, which can be up to ~64TB using the maximum of 32 extents) – but rather almost 2TB.   In fact, it’s 2TB – 512 bytes – or specifically 2199023255552-512 = 2199023255040 bytes.  Huh – that seems like a nit? 

this would be totally silly EXCEPT that I think a fair number of people will bump into it because most arrays enable you to provision LUNs in GB or TB, and people will in some cases say “2TB please” – not “1.99TB”

Watch this video if you want to see this error in action:

So – here’s the scoop:

1) 2TB is a hard limit (still) in vSphere just like it was in VI3.5 for a single partition – but VMware is now actually stopping an out-of-space condition that can result in a bad case.   If the last 512b is written to, bad things can happen.   This change eliminates this possibility.

2) Note the behavior with LUNs that are LARGER than 2TB – it used to be (in the 3.0.x days) that you could get borked by simply saying “use all the available space” and bam, your VMFS datastore was gone – poof.   See this KB article.   You can’t bork yourself here – but you CAN waste space (after all in the example I did in the video with a 2.5TB LUN, only 500GB could be used), and who wants to be inefficient?  

Remember – you can absolutely create large VMFS datastores, and extents can be your friend – read up on this here.

Back to the lab – it’s a lot more fun than watching all this DDUP stuff :-)