Virtual Geek

Get your SDS here (form to access ScaleIO)

Lots of cool new tech to play with when it comes to software-defined storage models.

Recall - SD* is about:

• Decoupled control and data planes (for compute/network/storage) - and control planes (which control policy) that are pure software.    The control plane abstraction enables abstraction, pooling, automation.  It's what ESX does for compute, NSX does for network, ViPR does for storage.
• Data planes (in storage land - these are the things that persist data and offer data services) that increasingly - wherever they can - are implemented as pure software (and therefore offer different deployment models - software + COTS or Software/Hardware appliances).
• Everything being driven programmatically via APIs.

Things like VSAN from VMware (great for all-VMware use cases), or EMC ScaleIO (targets larger scales - up to 100, even 1000's of nodes - and multiple different use cases - including VMware) are examples of #2.  

I'm seeing more, and more customers (including some big ones - see AT&T's view here - which is focused on SDN, but trust me, they are applying the same pressure on the storage domain) that want software + bring your own hardware models - which is exciting!

• EMC Scale IO is currently GA (v1.0) - and the 1.2 beta (more data services, specifically snapshots, native windows support on top of linux) is open!  You get access via this application form.    APPLY!!!!
• VSAN is also in a public beta, and VMware has stated a "1H 2014" target.  You get access via this application form. 

…   And, we are counting down the hours to the ViPR GA date (which will be the industry's first software-only, open, accessible, extensible storage control plane).

Fantastic esxtop post on VMware communities

If you're like me, you spend a fair amount of time looking at esxtop data.   There's a really great post that just went up on the VMware communities I think is just fantastic if you are looking to interpret results and wanted to share!

http://communities.vmware.com/docs/DOC-9279 

BTW - Happy new year to all - and here's hoping that 2009 is filled with peace, happiness, surprises, and interesting challenges for us all!

Scads of posts I've been contemplating over the holiday, chomping to get back in business!

VMware View Composer/Linked Clones - they are NOT a panacea

May seem strange to some of you for me to post this RIGHT after I do a post on the VMware View Reference architecture :-)

It's not, however.   There is no IT panacea.   There is only technology to be applied to problems.   And, generally, I think it's about know-how, not technology itself.   Don't trust people who say otherwise :-)

Ok - what are we talking about?   Ok, got a question from a very advanced VMware shop today looking at this and examining the blogosphere for info.    OK, background reading...

First - a good post on this here from Stu at vinternals.

Second - a good rejoinder/ongoing commentary here from Rodney.

Take a read - then come on back and read on...

Does VMware DPM shorten ESX server lifespan?

The inimitable MikeD (http://www.mikedipetrillo.com/mikedvirtualization/) pinged me and some of my colleagues this week with this question - he was starting to hear it, thought it sounded like FUD.   The question is:

"Does the Distributed Power Management power-on/off cycle cause the boot LUNs of the ESX servers to fail prematurely?"

BTW - if you're not a DPM expert, this video highlights it's incredible promise (it's currently an experimental feature in VI 3.5).

The answer in a word?  NO.    DPM does not shorten ESX server's lifespan due to drive failure.

Ok - two comments.   The first is that FUD is always funny (sometimes even our own - and yes, I know sometimes EMC is the FUDer not just the FUDee - though I try as hard as I can to not do that).   The second is that there is more than just the one word answer of course, otherwise it wouldn't be another long-winded Virtualgeek post, would it? :-)

If you're interested.... Read on!

Why I love ESXi

I've been tearing my hair out with a future build of ESX for a couple of weeks.   Now my system is decidedly NOT on the HCL.  It's one of my home-grown ESX 3.5 u2 systems - an Intel G33 motherboard, Q6600 CPU, 8GB of RAM, bunch of Intel 1000 Pro GT NICs.   To boot, I've been using an old-school QLA4010c (thanks qlogic for sending me the new QLE4062s and the cool 8042 10GbE converged adapters!).

OK - this system boots FINE under 3.5u2, but under... let's say... ESX.next, not so much.   NOTE - we're talking pre-release here, so don't extrapolate ANYTHING from this.

• During install, it would hang while loading the network drivers - I eventually isolated this to the Intel GT cards.   PTs, OK.   MTs, OK.   Interesting.  ?If I don't disable the onboard NIC, same deal - hang during install.   They all use the e1000 driver, but mental note - I'm using MTs and PTs going fwd.
• the internal SATA controller looked like it installed fine, but then looked like the MBR wasn't configured right - at the end of install, the HDD wasn't bootable.   I found this long ago with non-supported controllers, so poked around with this.
• So, the QLA4010c (and this seems to be true with the QLE4062 too) boot from iSCSI SAN wasn't working right (BIOS loads, drives found, but don't show in the install screen).

I poked, prodded.  I tore my hair out.   I was stuck with no bootable controller/drive combo, and was considering a trip to Fry's for another MB whose SATA controller I would know would work (an older nvidia MB for example).

Then, I had an idea. 

I've used a bootable USB ESXi 3.5 image on my laptop, and suddenly thought - "why don't I just do that?"

A few minutes later with DD for windows and the dd image I had my ESX image on USB.

BTW - The always excellent Duncan's got a little post on how to build your USB key here: http://www.yellow-bricks.com/2008/07/29/esxi-35-update-2-on-a-usb-memory-key/

So, why am I so pumped on this?   It's more powerful than immediately apparent.

1. Simple.   The process of dd takes a few minutes, and configuring the system after boot is short and easy - faster and easier than the installable image
2. Portable.   I'm carrying it around with me now and am doinking around with ESX.next on the road on my laptop.
3. Compatible.   All those driver issues I was struggling with?  NON EXISTANT this way - I've got my iSCSI HBAs, my NICs, and I'm up and running.

Nothing earth shatteringly new here, but rather a real "aha" personal moment where the power of this technology and approach. 

I'd strongly recommend that anyone out there who manages a VMware environment play with this.   If you're used to installing systems off DVDs, you might just find yourself considering a new way.

SVVP - how does it work?

There's been some good discussion around this today, and it was good, tomorrow am some official language will come out that will be along the clarifying lines in the body of this post.   Remember, just because I post it doesn't make it official, it's not official until it comes from VMware.   But, this should elucidate things a bit.  

Ok - open standing proposal...   First some history...   EMC has a long experience in working specific customer configurations through the Windows Server Catalog.   Getting MSCS configs certified historically (it's recently gotten a lot better, thank goodness) was BLACK AND WHITE.  Only that EXACT configuration was supported.   For many people this was a deal-breaker.    Over time we got really good This turned out to be a good thing for MSCS as a whole, when you're building these mission critical config, support testing    We became a machine at doing this.   Want a fun proof point?

Step 1 - go here: http://www.windowsservercatalog.com/

Step 2 - click on "cluster solutions" - note there are: 6,792 items found

Step 3 - type "EMC" in the storage field and search: 3,380 items found

You can see that we've done this A LOT - just about half the MSCS configs are on EMC.

OK - here's the open offer - if you are a customer, and the only thing stopping you from adopting a joint VMware/EMC solution for your Microsoft apps is a specific gap on the SVVP that matches you, post your needed config as a response to this thread, I'll work with VMware to get it up there. 

I'm that sure that virtualizing wth VMware is the right thing for customers - I'll get YOUR config tested.   We do have experience on getting this sort of thing done fast.

Ok - for some Q n A - read on.

HOWTO - Benchmark an iSCSI target (and any block target, including within VMs living in NFS datastores)

Someone asked me out of band  (thanks for asking, MattL) what I would do to quickly get a envelope of an iSCSI target.  

In all the post-VMworld followup, I don't want to do a long dissertation here, but quick off-the-cuff comments may be useful.   FYI, these are the quick set of tests we do during the application validation testing to quickly see if we're on the mark before we start with app workloads.   Read on for the details!

So, what does vStorage really mean?

(UPDATED 092308 - 19:53 EST to note that the VMware Backup Framework falls in scope)

Pretty big day here at VMworld, meeting lots of people I know only online - which is great.   My photo is in the about section of the blog, and if you'd like to meet me, I'd love to meet you!

So - the VDC-OS is the big news, let me fill you in on one part of it - vStorage.

vStorage has been something that VMware and EMC have been working together on for a long time, in fact, before it was called vStorage (which was recent) - it used to be called VMAS a name only engineers would like.  In fact, we've been working on this from almost right after the original acquisition before the program even existed formally.

vStorage is pretty broad: It's 2 very new VMware things, and 2 VMware features

• vStorage API for Multipathing
  • This was the first idea from eons ago.  Customers know the pain VMware multipathing can be - setting up static paths, manual "load balancing" by rotating primary paths, and no automated path discovery.... you know what I'm talking about.... But we needed to find a way that we could get PowerPath, which is the far and away the leader in this space, into the vmkernel, but still make it open for others to also innovate with VMware.  It so happens of course that EMC are the ones way out in front here.
  • So, while native multipathing (NMP) continues to get better (round robin will likely soon no longer be experimental), but this gives EMC customer another option - an option which is a quantum leap in multipathing/failover/scale and availability for block storage
  • It's also an avenue to deliver the other stuff PowerPath customers love in the physical world (path-based encryption, accelerated data migration)
  • It's also simple, and easy to use and install.   It just works, like the best technology does.
  • BTW - we're accepting beta sites now, and it's going to be HUGE for any VMware/EMC customers.

See this quantum leap in multipathing action here:

download a hi-rez version here.

• vStorage APIs
  • Acceleration via storage offloads using SCSI driver primitives (we're demoing this in our booth and in the VMware/EMC/Cisco Keynote) - think "I/O" dedupe (not to be confused with Storage Dedupe).  This can have a HUGE impact on storage network utilization - much more efficient, and very important for lower bandwidth storage networks like 1Gbps iSCSI.
  • Thin Provisioning Integration
  • While NetApp, EMC, and Dell/EqualLogic are all shipping Snapshot VC integration today (we're demoing Replication Manager in our booth and in the VMware/EMC/Cisco Keynote, and you can also see a demo of this here), vStorage will make this more transparent, but adds somthing REALLY cool - the ESX server tells the array what BLOCKS makeup a VM, so svmotion can be offloaded to the array as an alternative to the file-copy method now, snapshots can potentially be done on VMs even for block devices.  Again, this uses SCSI driver primitives

See this the "I/O dedupe" in action here:

download a hi-rez version here.

Also as part of this, some things that have existed before are getting clarified.

• SRM's SRAs are getting positioned more clearly now as vStorage APIs for Site Recovery Manager (since they are in effect, APIs for VMware/Storage integration)
• VCB is more correctly positioned not as an VMware product per se, but as a API framework for backup vendors - becoming the vStorage Backup Framework
• vStorage Virtual Appliances now formalizes a program around VMware Ready appliances that perform storage-related functions.
  • Celerra VSA freely available – iSCSI, NFS - public and free!
  • Avamar Virtual Edition – shipping for 1 year

Cool stuff we've been working on to help customers, I'm glad to share the news!

UPDATED - 9/13/2009 - What we're demoing at VMworld - details!

Some days, my head feels like it's going to spin right off :-)  Thank goodness I love my job!   I'm personally working on some of the last-minute demos - there's some really cool stuff that's under embargo until Wednesday - but suffice it it will be cool!  

My colleague Nick Triantos from NetApp did a good post on what Netapp will be doing, here: http://blogs.netapp.com/storage_nuts_n_bolts/2008/09/netapp-vmworld.html?cid=130026414#comment-130026414

I have to confess, it got my blood boiling a little bit - he accuses EMC of being "All Marketing" and that they will be the largest vendor there.    Please, look at my record, I've been kind, even handed, even complimentary to NetApp, but come on Nick, is that condescending tone necessary?     Tar ME with the marketing brush?!?

Did you know that while we're mud slinging (well, at least I'm getting mud slung at me) that the Large Hadron Collider fired up today?   Man, it's inspiring, and a reminder that this competitive stuff is SO petty.  I'm also pretty inspired by the SVVP "other shoe dropping" with formal ESX 3.5u2 qual getting done.

Ok, so we have a marketing gimmick (the car!!!), sure but heck, it's Vegas :-)   I like to think of us as "All technical, and some fun too"

You'd think that with all this back and forth - it's all we do (well for me - it's 8:51pm, and I've been working all day - so this is a diversion :-)

I will tell you this, it's going to be technical, we're demoing like mad, and there are some MIND BLOWING things there.

We're in booth 500, and I look forward to seeing you there.   What you can expect to see:

• Demonstrating Oracle 11g and SAP with instant/backup recovery and Site Recovery Manager all in action together  (BTW - thank you all who have been so interested in the Oracle piece that it got oversubscribed and we've been asked by VMware to schedule a second session on EA1961)
• Demonstrating Exchange 2007 Joint VMware/EMC reference architecture including VSS backup/restore, CCR and LCR use cases
• How all our management tools (Storage, Network, Application Dependency and VMware Best Practice compliance analysis) integrate with the VI 2.5 SDK and ESX CIM APIs - TODAY.   Our view is that customer need to start virtualizing tier 1 apps - and we want to help - with the joint best practices and with tools to ensure you get applicaiton-level SLAs.
• Replication Manager for VMware - instant VM or datastore backup/restore - integrated with the VI and ESX APIs for nice simple point in time consistency.   As a company that's not as good at marketing as others, we quietly actually shipped this BEFORE ANYONE ELSE.   It wasn't pre-launched, it shipped in June.   Here's  a quick look if you want to see what it does.   BTW - this is a single tool that does this for application integration Exchange 2003 and 2007/SQL Server/Oracle and now VMware, and across all EMC's platforms.   Here's a demo....

• VDI - great to hear that NetApp hit 5K clients. We just broke 10K :-)  UPDATE: I regret phrasing this sentence this way - it comes across as snide and as condescending.   I let frustration get the better of me.   NetApp has no intrinsic scaling limit that I know of, I only know what we can do.   BTW - that's what this standing lab in Santa Clara is there for - for EMC and VMware to jointly prove these out at scales (target of 40K clients).    BTW, we're also doing this at 500, and 1000 client cases.   It's not a chest-beating exercise - customers come in all shapes and sizes, and we're trying to help prove out how this works for them all, it was triggered by the World Bank which asked for us to show how this would work for 20K clients.      We have a similar (but different - each with advantages and disadvantages) approach as NetApp - mass scale with consumption of one, both how we do this today, and also how we do it with VMware tomorrow.    We will be sharing all our performance scaling data, and experiences.   THIS IS BLEEDING EDGE.   Anyone who tries to tell you that this is all figured out is not giving you the whole truth.  Anyone who just shows you the array side of the equation is flat-out dangerous.   When you push this far,  and all boot simultaneously, sometimes VC gets into a weird state on some of the VMs.  Its fun, and continues to be fun.   BTW - here's a cool screenshot (count up the VMs - there's actually 10,994)

UPDATE: Folks, the NetApp side has made some comments that in their view we're not being clear here: http://blogs.netapp.com/virtualstorageguy/    Vaughn is a good guy, so I'm going to take the personal slag (ouch - used car salesman) in stride.   We are replicating the LUN objects in this case, and they are the unit of replication.   If you will have more than roughly 200 VDI VMs per cluster, you need to have several VMs per LUN.   The PowerVDI tool automates this.   These ratios mean it's more accuate to say "you will invariably achieve 100:1 space savings, and may achieve more depending on the scaling goal". (the ratio of source VMs per source LUN needs to be higher at higher scales, and therefore 100:1 is the worst case model).   There is NOTHING intrinsically wrong with NetApp's approach, there is NOTHING intrinsically wrong with EMC's approach.   Each have advantages and disadvantages, and are a function of each vendor trying to solve the customer problem.  The ratio of LUNs to VMs is not 1:1.   We don't do this because we love block - while we can take 96 filesystem snapshots, only 16 of those are writeable, so this approach is better for EMC customers. Just setting the technical record straight.

 

 

• Here are some datapoints from the VDI testing to date:
  • there are limits to how fast ESX and VC can register vms (hostd dies) and how many you can spawn simultaneously - discover the what we've done jointly with VMware on this!
  • XP Booting causes 4000  IOs spread over 20 seconds.   That is 200 IOPS per Client Sustained during the boot….   Multiply that times 100 for a server…… 20,000 IOPS.    Or multiply it out to a moderate environment of 1000 desktops.     200,000 IOPS all directed at the Array.    How much hits the backend vs cache effect?   What is the client experience?  Come and find out!  (BTW this will be a big topic in session PO3824 - "Storage Solutions for Enterprise Consolidation with VMware")
  • BTW - Congratulations Dan for coming in 3rd in VMware's PowerShell contest for the PowerVDI tool we created for this.   The quote was great: "Here's what Lee Holmes, author of the Windows PowerShell Cookbook had to say about it: "Automates a very complex scale-out task, and combines a lot of technology (VMWare, Putty, AD, etc) into a single PowerShell task. Offers interactive use, unattended scripting. For high-scale clusters, this would be an enormous time-saver. This script really demonstrates the super-glue nature of PowerShell."   Dan, you'll laugh man, but a large NetApp customer asked me for the script... wait until we unveil what's coming next!   For those of you wanting this script, note that we've posted it openly for all.
• Deduplication, and simple file-level recovery for Windows/Linux VMs
• NFS and iSCSI deployment best practices
• Joint EMC/VMware Remote/Branch Office solution
• Several of our Partners (Agilsys, Fusionstorm, ICI and others) will be present in the booth to show their integrated solutions.
• We will be demonstrating several things I can't even allude to as they are under an embargo until the event.

We will have 70 people at the event, including myself, but also our engineering teams directly - including Bala Ganeshan and Sheetal Kochavara who are the engineering folks who maintain our DMX and CX best practices, along with our Field VMware Specialists (all VCPs from around the world.  

I don't know what any other vendors are sending equipment wise, but I've sent 13 CX3s, 4 CX4s, 3 NS20s and 2 DMX4-950s.   Those will be in lots of places where you can use them, including the hands-on labs, and in various booths.

I can wait to talk to YOU, and of course with our competitive brothers and sisters!

Oh, and our keynote  (where all are welcome!) is KN EMC on Wednesday at 11:00-12:00. It is a joint VMware/Cisco/EMC session discussing best practices for designing the next generation datacenter with Scott Davis from VMware and Ed Bugnion from Cisco.    It's three nerds on stage - a recipe for trouble!      I asked to do a show release to show what we're doing at the event, but our PR folks told me that unless a senior exec is doing the Keynote (like Dave Hitz is for NetApp - and they did a press release), we don't do those.  I'm just a technical guy, so I don't qualify :-)

Just to show the ridiculousness of it all, NetApp's press event got scheduled at the same time as our keynote.   We intentionally moved ours to avoid conflicting with theirs because we know some people will want to go to both.   I'm going to keep pushing the kind agenda, because I don't think pettiness helps any of us.

Nick says:

"For NetApp this will be the largest event of the year and like everybody else we've been preparing for it for some time now. This year, by far, we'll have the largest presence in the show and we'll be ready to showcase various technologies that solve some very real and hard to solve business problems."

UPDATE: Nick has made a fair clarification of his intent  of the statement, and I trust him in his clarification.

Nick - looking forward to meeting you there and getting to know you.   This is our biggest event of the year also, a bit bigger than EMC World which had 10K customers.   I don't know if we'll have the largest presence at the show, but I can tell you that you are not in a position to make that statement that you will be until after the show ends, right?   Statements that are not true?   I call that Marketing.

.

.

.

OK, I can't help being a bit petty.

Remember the whole brouhaha about the IDC data back here?  

So, on Sept 8th, Goldman Sachs released their annual survey analyzing IT spending - the "Independent Insight"  here's how THEY describe it.

"This is the 42nd issue in our IT Spending Survey series. Our survey panel is made up of 100 managers with strategic decision-making authority at multinational Fortune 1000 companies."

Ok - so note (I don't want to misrepresent!) note that this is FT1000 (we're doing excellent in all segments - but I don't want to imply anything about other segments, take it what it is, no more, no less)

First, for my VMware brothers and sisters - congrats:

"VMware’s incumbent position at the top of the rankings demonstrates that server virtualization momentum remains alive and well."  (Microsoft - watch out for the wrath of Nick, showing Live Migration when Hyper-V R2 is scheduled for a 2011 release is called Marketing)

Nice to see EMC Software (which excludes VMware) also in the gainer column - our management tools are flying off the shelves, in some part due to their integration with the VI SDK and ESX CIM APIs.

I liked this, of course:

Does that look like it agrees with the IDC data from earlier to you?   Look also at who went up and who went down.

Oh, and then here's this hot off the presses too: http://www.idc.com/getdoc.jsp?containerId=prUS21411908   The NAS numbers and growth numbers are stunning.

LET ME BE CLEAR: I THINK NO CUSTOMER SHOULD PICK A SOLUTION PURELY BECAUSE OTHERS DO.   Every customer should pick solutions that work for them.   What I am saying is you can't produce those results and outgrowing those that are smaller, and do it across categories if you're not doing something right for your customers.   To believe otherwise is to shrug off reality, and to become one of those "extremeophiles".

Look, I want us to focus on helping the customer.   Nick - I'm good with that if you are - sincerely......

I will be in some NetApp sessions (and in VMware labs, and in other sessions I'm interested in).   I will not heckle - though of course you guys are welcome to in my session (leave my colleagues alone).   I'm respectful and want to learn from you and with you.   But I won't take mudslinging at EMC lying down - in the same way that you guys shouldn't (and don't) when we do it.

Here's to seeing YOU at VMworld 2008

As we get in to the final stretch of preparation - I'm really hoping to see you there.

Here's a hint at the EMC contest (come by the EMC booth for more info - we're booth 502):

I really hope you'll come to all our sessions, we're trying our best make them pertinent, technical, filled with demonstrations, disclose current and future capabilities, and share customer deployment experiences. 

In the booth, we'll be distributing best practices and virtual appliances you can use, demonstrating solutions from VDI to SRM, Management integration, powershell scripting, VMware integrated VM-consistent array replication that simple and easy to use, and loads of other fun stuff, some of which I can't talk about until Wednesday.

My session is KN-EMC, on Wednesday at 11am.   It's a technical panel discussion on overall Next Generation Datacenter design and best practices, and I'm joined by Scott Davis from VMware (Chief Datacenter Architect, CTO Office), and Ed Bugnion from Cisco (VP/CTO, Server Access and Virtualization Business Unit)

If you haven't yet registered, it's getting close, but you can still squeak in - click on the graphic below.

FYI - I've posted my current calendar (I arrive on Saturday and leave Friday), hope to see you in some of the sessions!   if you want to see my schedule, read on!