Cisco’s Own Megalaunch – Details and Perspectives

Cisco today launched a wide-ranging set of updates to their Datacenter Business Advantage portfolio.   To me – this is like EMC’s own Mega-Launch in January.   Like us (in January, we refreshed/launched 41 products all at once), they have updated not one product, but a whole portfolio at once.

For what it’s worth (I’m just one voice) – color me impressed.

Like the January launch for EMC, the sheer fact Cisco can update so much in a single go, cover such a broad spectrum…   Well, it highlights the breadth of what Cisco has become, and is in the process of becoming.  On the note of “what they are in the process of becoming, see the NewScale acquistion as just one more example – strange in some ways, but to me makes a TON of sense (analagous to EMC’s Greenplum acquisition, or VMware’s Spring/vFabric – both reflecting possible answers to the question of “where is the puck going?”)

So – what’s the news?   Don’t consider this an exhaustive list – but here are the things that tickle my nerd funnybone.

• New Data Center Network Manager (that ties to broad NX-OS revs)
• Nexus 7K FCoE support.
• Multihop FCoE across the whole family (N7K, N5K, MDS 9K)
• A new Nexus 3K platform – interesting in the sense it seems to be designed for a very specific use case (very low-latency use cases).
• Crazy updates to the N5K – with more dense configs (5548P and 5596UP), but also “any port, any protocol” design (better than the “FC module” approach, IMO).
• New Cisco UCS servers – refreshes to Westmere-EX across the B-series family, addition of a new C-series (C260M2)
• Cisco Storage Media Encryption (Fabric Based Encyption)
• New Fabric Extender (FEX) modules – very interestingly including host adapters
• Location/ID Seperation Protocol (LISP) support on the N7K – this is a very interesting idea (more below)

The launch is so big that there are many other things that while interesting, are more “meh” (though I’m sure important to customers) – like Layer 3 MPLS support on NX-OS in the N7K.  This is important as Cisco works down their own convergence path.   It’s interesting to look at the evolution of both Cisco and EMC – we both are working to converge where it makes sense to converge, but simultaneously, accelerate expansion and diversity in other areas.

First of all – to my colleagues at Cisco… CONGRATS!   Trust me, I know what it takes to pull something like this off :-)

Will try to provide more details, perspectives (as well some performance testing, and EMC E-Lab support targets) in an even-handed way.   If you’re curious, read on – and as always, courteous comments welcome!

OK – let’s take these one at a time in more detail:

• New Data Center Network Manager (that ties to broad NX-OS revs)
• This is a new product, one focused at getting the “big picture” of the converged network, and I’m happy to see that it has tight vCenter integration (again, analogies galore – this is like our VM-Aware Unisphere – and future VM-Aware SMC – embedding vCenter API integration and VM object awareness/correlation directly into the infrastructure itself).
• This thing has a pretty darn big scale (150,000 ports)
• I haven’t tried it myself – so I can’t comment about the user experience (though I will be trying it).   While I’m always skeptical about gen 1 management products (this applies to EMC and VMware too), they represent perhaps the more important capability that our “core” products.   Good to see big investment from Cisco there.

• Nexus 7K FCoE support.
• This is perhaps the biggest part of the release.   In my experience FCoE is having at best what would be characterized as moderate adoption in the market, though 10GbE adoption is now relatively fast and furious.   I think people are finding (like I found with iSCSI when at an iSCSI startup) that FC is surprisingly difficult to displace.    Where it is finding success (at least in my experience) is in:
• New, greenfield host-to-top-of-rack switch configurations.
• In tightly coupled UCS use cases (a different manifestation of the same point).
• One view (one I’ve heard from customers) is that they won’t go down the converged network until they can see how it will carry through the core, not just access.   Remember that the question of “how do I get to a converged network” is more important than FCoE per se.   You can have a converged LAN/SAN without FCoE (using just iSCSI and/or NAS), but you CAN’T have a converged network with 10GbE and FC maintaining separation.   This announcement gives Cisco customers a clear access/core converged network option.
• To put this in perspective (all using list prices):
• Imagine you assume a “baseline” of the cost per host attach using a Catalyst 6500 coupled with MDS9500.
• Deploying a N7K with SFP-SR optical modules is %18 lower cost per host.
• Deploying a N7K with copper twinax (limited to 10m distance) is %45 lower cost per host.
• On top of the all the other elements (space/environmentals, manageability).  

• Multihop FCoE across the whole family (N7K, N5K, MDS 9K)
• The NX-OS update is a big, broad one – with many new capabilities, but one that’s worth calling out explicitly.  There have been LOTS of debates and discussions about whether this was needed, and where/how.   I’d highly recommend reading this (by Scott Lowe) and this one (by J Metz), this one by Brad Hedlund, and this one by HP (to be fair and balanced).    Why do I think this is important? Because, while all the vendors were fighting over semantics – customers were stalling.   FWIW, while I think it’s fair to call a FEX a “hop”, it missed the point.  Customers wanted to see how they could (and if they should) create access/core architectures and had basic expectations about how it should work – Cisco’s answer is now relatively clear.
• The MDS9K getting an 8-port 10GbE FCoE module (pic below) is also a really big deal IMO.   While Nexus is moving fast, there are TONS of MDS customers around the globe.  This gives them an easy, lower cost way to start their own access network convergence journey.   If it were me personally as a customer, I would still push hard to go the N5K route (as it’s clearly the longer term direction), but it’s good to see Cisco giving their existing customers that lower-barrier choice where they can get more leverage out of their existing investment.

I’m going to “break” here from the flow of analyzing the Cisco launches and present some very early data.   EMC and Cisco have been working together for a decade plus, and it’s hard to understand just how much work EMC E-Lab and Cisco have been doing on FCoE and converged network testing.   So…  Check this out.  We did a bunch of tests with the N7K (native FCoE) and also bridged FCoE (going through a MDS9500 with the FCoE module) going into an EMC VNX with native converged 10GbE interfaces.   We used UCS servers and VMware View to generate some load (using 100 Win 7 clients)

Here’s the FCoE port properties in Unisphere.

And here are all the FCoE initiators logged in.

So – what did we find? – if you look at the results below, the performance envelopes were nearly identical.

More data will be presented on this over time, but I wanted to give a “sneek peak”… Back to the regularly scheduled analysis….

• A new Nexus 3K platform – interesting in the sense it seems to be designed for a very specific use case (very low-latency use cases).
• This one is funky, and an oddity, but I can understand where it’s coming from.   Have been spending more time lately with customers in these strange ultra-low latency use cases (finance, HPC).  Have also been spending time with Isilon (who uses Infiniband for certain things).   While Infiniband has never exploded on the scene (for whatever reason – IMO the use cases are just to narrow to trigger the mass adoption curve), there are use cases where latency is EVERYTHING.   Another example is the VMAX RIO memory interconnect.  There, latency is so tight that we still think that IB has a little way to go.
• The interesting part to me is that Cisco is still targeting 1us (microsecond) port-to-port latency.   Infiniband today is at 100ns (nanoseconds) – a full order of magnitude lower latency.  It will be interesting to see what the customer reaction is – in these absolute low-latency use cases, will they go for “very fast, good enough Ethernet”, or “look, I need the lowest latency I can get, forget everything else”.
• Crazy updates to the N5K – with more dense configs (5548P and 5596UP), but also “any port, any protocol” design (better than the “FC module” approach, IMO).   This includes 8Gb FC support.
• Bigger/faster/stronger, yes, but there’s more… (pic of 5596UP and 5548P below)

• The “big news behind the news” on this one to me is that Cisco has committed that the N5K platform (not clear if it’s just the new ones) will be getting FabricPath support (post FCS – which means “in the future”).   These “new flat networks at scale” is likely the next big battleground between Cisco (FabricPath) and Juniper (QFabric) and Brocade (Virtual Cluster Switching) – so, to Cisco customers, knowing that there is a path there is important.
• One little “weird bit” is broader 1GbE support – surprisingly, this in the past has been a snag for some of the folks on my team, and I bet we’re not that unusual.   Good to see “1GbE on every port”.
• The other very interesting news (again – to me), is the addition of a fair chunk of L3 services to the N5K, via an L3 services daughtercard.   Not a surprise (something obviously needed), but in some ways, to a smaller company, in a smaller launch, this might be PR on it’s own.   While this won’t substitute (again, IMO) for any significant router, it adds certainly “good enough” L3 services for a TON of use cases.  Pic of the service module below… 

• New Cisco UCS servers – refreshes to Westmere-EX across the B-series family, addition of a new C-series (C260M2)
• Cisco has been doing a KILLER job at being very fast out of the gate with Intel CPU/platform support.  Seriously, even if you’re a “Cisco hater” (and strangely like “EMC Haters”, they tend to be pretty vocal) you’ve got to give it to them – being as fast as they have at synchonizing to Intel’s roadmap and deliverables is impressive.
• One thing I find customers don’t know enough is that there’s now a lot you can do with UCSM with the Cisco C-Series (rackmount) that you couldn’t at the beginning.   The C-Series are interesting.    You may not think of this immediately, but EMC sells a LOT of servers (literally enough volume that if we called them “servers” as opposed to “appliances” we’d be on the list of larger server channels.   Every Recoverpoint, every Atmos, every Centera, every Avamar grid (etc – it’s a long list), is in essence an OEM’ed server, running EMC software.   As the industry shifts around, will be interesting to see what (if) we do with these C-Series systems.  They are solid, they have a lot of the UCS value.  Cisco added a new one – a 2U server, 2 socket, 64 DIMM server.   Yikes, that’s a lot of DIMMs.   I’m interested in getting some quotes so I can see pricing.

• Cisco Storage Media Encryption (Fabric Based Encyption)
• Data-in-flight and Data at rest encryption is increasingly becoming very important to customers.  In certain use cases and verticals, this is becoming an RFP requirement.   Strangely, for a tech that in some ways is relatively basic (on it’s surface, “how hard can line-rate encryption be?” seems like a legitimate question).
• When you dig a bit deeper, I think there are reasons why this hasn’t just become a “yes, it’s there” answer across the entire market.  The first part is that the technical problem is actually harder than it looks at first glance.   The first reason is that in storage/network use cases – latency matters so much.   Encryption, even little additional bits of latency hurt.
• The pure “just do it at the disk media” helps with a narrow set of use cases (stolen or misplaced disks), but with none of the others.
• Another reason (and I’ve seen customers suffer horribly on this topic – thank goodness not EMC customers, not to say it couldn’t happen to us) is that any issue with robust key management with any of these approaches (host based with or without HBA assist like the Emulex/EMC Onesecure solution for midrange customers, fabric based, array based like the EMC VMAX solution) – you can have a really bad day.    Even a momentary inability to access the appropriate keys leads to what EMC calls a “DU” (Data Unavailable) event – which is very, very bad.   Even worse, loss of the keys leads to what EMC calls a “DL” (Data Loss) event.   This is the core “quid pro quo” of data encryption.  So key management is everything.    I’m glad to see that Cisco and RSA have teamed up – and that you can use native key management, or integrate with RSA key management.

 

• 

 

 

• New Fabric Extender (FEX) modules – very interestingly including host adapters
• This is an interesting architectural direction Cisco is going, and I don’t know whether people sufficiently understood it’s gravity early on.   When UCS was “Project California” and I was introduced to the FEX idea – using VN-Tag - for the first time (this was several years ago), I certainly didn’t get the import.  At the time, it looked like a simple, elegant (but new and funky) way of extending a switch in every logical way to multiple chassis elements (which is how it’s used in UCS).
• Later, I saw it’s “second formulation” which was the Nexus 2000.   Here, it could extend a ToR switch to have greater density, greater reach, while keeping things simple.   Here you had: for some smaller customers – a “flattened network" in a simple way; and for others, a simplified access layer.
• Clearly, this was better characterized as a “foundational” technology, rather than a point product.  So, today, there are a couple parts of this launch that continue to find new uses:
• The first is less funky, but simple and practical – you can now use Nexus 2000 FEX with Nexus 7K.   This is interesting in the sense that you can get the “ToR” cabling and management with a modular chassis core while still having a “flat” design.   This also means new crazy port densities (1024 x 10GbE ports).
• The second is more funky – there are now “Host FEX" modules with VM-level visibility.  While not a technical revolution (looks like for all intents and purposes, Cisco is making their Palo-based adapter available as a standalone HBA).   This is VERY interesting to me.   This interface virtualization (including VM-level direct mapping) is a core feature that UCS customers dig.   This means that to some extent, this can now be available to other servers.   It also means that while there’s a good debate to be had about MR-IOV vs. the VN-Tag + interface virtualization approach – it’s going to be very very hard sledding for the various IO virtualization startups.   This is commoditizing fast, faster than I would have expected.
• Location/ID Seperation Protocol (LISP) support on the N7K – this is a very interesting idea.
• OK – I remember when I did my first cellphone carrier swap using Local Number Portability.  I was very, very happy to be able to switch from AT&T to Verizon and keep my number.  
• Now, think about it – IP addresses are tied to physical devices.   That makes “IP portability” basically impossible.    There are things to “stretch” L2 networks.  There things that “tunnel” L2 over L3 (OTV).   But, there’s something else needed – “IP portablity”.   If you think about it, the IP address represents two ideas in one – who you are (the device itself), and where you are (how to route to you).   LISP basically uses two IP addresses – one to represent how to get to you (“global meaning”), and then the second is who you are (“local meaning”).   The upside of the implementation is that it doesn’t affect the end-node (it doesn’t need to know about anything beyond it’s IP address).
• The news is that the Nexus 7K supports LISP.  This has a couple big potential impacts:
• Workload mobility – moving IPs around in different use cases than OTV and other DCI use cases – more towards the “Hybrid Cloud” use cases rather than “bridging Private Clouds”.
• For service providers, being able to create tenant domains without crazy management difficulty.
• Making IPv6 deployment a simpler “step in function”.
• I’d highly recommend reading up on LISP – start at the IETF charter page here.

OK – so – what’s the scoop re: EMC E-Lab timing?

• The MDS FCoE Module and the Storage Media Encryption are on E-Lab day one.   For customers who choose to acquire via EMC Connectrix, that should be in the July timeframe.
• All the Nexus updates are still going through the last steps of E-Lab testing, and should expect another 2 months or so (partially coupled to final product availability from Cisco), but everything is looking good in the early stages.  The folks at Cisco are pros.

To all the Cisco folks who have worked SO hard on this – again, my congrats, and celebrate!   To customers – would LOVE to hear your feedback (critiques, perspectives, what you are doing – all welcome!)

Comments

[Flash Mob]

I was shown a Cisco roadmap which shows this FCOE blade as an FCOE-target only solution, and says "no LAN support". How is that a convergence solution? Also, it only has 8x 10GbE ports. Why just 8? I saw Brocade has had an FCOE blade that has 24x 10GbE ports and does support LAN connectivity. Let's see, if convergence is important to me, would I prefer 80Gb of bandwidth or 240Gb of bandwidth? And yes, LAN connectivity would be nice in a converged network. I have to say, the author sounded so hopped up on the Cisco kool-aide he seemed like he was about to pass out. If your an engineer, you better do your research before jumping into the converged networking pitch - there's a whole world of planning that is critical. My LAN guys and SAN guys are going to have to learn to communicate with each other and be friends. Any time a change needs to happen on the SAN side, they are going to have to notify the LAN team, and vice versa. When a CNA goes bad, who's job is it to call the CNA vendor, the LAN guys or the SAN guys? I'm not saying stay away from converged networking. I'm saying have a lot of PLANNING sessions with both teams in the room. Or else you'll find your CIO boss will have your head on a platter. Also, no matter how much redundancy you build into it, converged networking is putting all your eggs in one basket - is that really worth the cabling and infrastructure "cost savings"? I will say the new ethernet which is more lossless and deterministic looks promising for the LAN side, but the SAN side isn't broken - converged networking doesn't solve anything for the SAN side, and in fact complicates things worse. Still, when (not if) things go wrong, do you want all your eggs in one basket? Yikes. I'll stick with our current model, keep my LAN and SAN separate - but I do like the idea of upgrading my LAN side to the new ethernet.

[Juan Tarrío]

Hey Chad, can you elaborate on what's new about the Cisco SME? I haven't been able to find any details about it anywhere, and comments like yours or the one seen at http://thenetworkworld.blogspot.com/2011/03/cisco-data-center-revamp-cuts-across.html ("What's more, Cisco unveiled the MDS 9000 Storage Media Encryption fabric service, which offers secure media encryption for disks and tapes to meet security requirements for regulatory compliance") don't add a lot of detail. This product has existed for over 2 years, so what is exactly new?

[Kelley McGowan]

Chad,
THIS IS FANTASTIC! I have a couple of groups on linkedin and I would love (with your permission to post s link to this..This is exactly the real deal content that I am looking for. I hate marketing fluff and BS.
Please contact me if ok..I am KelleyMcGowan on linkedin and cant wait to hear from you!

[Sean]

Great Article!

Dont forget the 6500 line card updates

http://www.cisco.com/en/US/prod/collateral/switches/ps5718/ps708/data_sheet_c78-643759.html

[Richard Rose]

Flash Mob, the MDS 9500 8-port 10G line rate FCoE module you're referring to is primarily an ISL module which bridges Nexus based networks supporting FCoE to the MDS 9500 Fibre Channel SANs.
This preserves investments customers have made in their Fibre Channel SANs, even as they move towards converging their access layer using FCoE with Neuxs products.

For LAN+SAN convergence on host facing connections, you will be looking at the Nexus platforms, the Nexus 5000 and now the "director-class" Nexus 7000.

As for comparisons to the Brocade DCX FCoE module, this actually is limited to 8x8Gbps FC connections over the backplane, or 54.4-Gbps of data (8 ports x 8.5Gbps @ 8b/10b encoding). You'll get a full 80-Gbps of data using the MDS 9500 8-port FCoE module.

Also AFAIK, in-service software upgrades (ISSU) are not supported when this blade is in the DCX, a non-starter for any mission critical storage networks. ISSU is supported on any Cisco switch that supports FCoE.

Disclosure: I work for Cisco.

[Paul Valentino]

All great stuff but I wish you would have bolded and capitalized the following:
"The pure “just do it at the disk media” helps with a NARROW SET OF USE CASES (stolen or misplaced disks), but with none of the others."
IMO data-in-flight is where the true risk lies for customer managed or private datacenters that have adequate physical security measures. And when I say data-in-flight, I'm not referring to the FC traffic that is isolated to the DC but what ends up out on the IP Network. While the attack surface may increase for converged networks, I'd still focus my energies on protecting the ingress/egress points into the DC rather than creating greater latency by encrypting data-at-rest or FC traffic.