Virtual Geek

vStorage APIs for Data Protection and Avamar 5

UPDATED April 30th, 2010 @ 9:14pm ET (linked to WMV demo)

UPDATED Dec 1st @ 10:10am PST (minor updates/corrections – also posted the table in xls format based on demand)

“What is the best way to backup my VMware environment?”

Ah… There are some questions that are easy and short to ask, but are long to answer.   This is one of them.   But – the proliferation of various tools for VMware-oriented backup (including VMware’s own VMware Data Recovery as well as other 3rd party options) suggests that good vendor answers to that question have a lot of value to customers.

EMC Avamar has long been one of the most popular ways to do backup in the VMware space (with many, many customers of every size including VMware and Cisco themselves – each with thousands of VMs).   The reasons for it’s popularity in this context are the following:

• simple backup and simple single-step restore in all cases (not just one or two guest type)
• good application integration
• saves a ton on storage due to data deduplication (commonly 30x-50x)
• dramatically faster backup due to the fact that the dedupe occurs before the data is transfered (aka source-based dedupe – commonly reduces network load by 300x-500x)
• higher server and VDI consolidation ratios in dense VM configurations due to lower LAN use during backup
• great for remote office backup (particularly where the remote office is too small for the smallest Data Domain platform) due to lower WAN use for centralized backup

What people didn’t like about Avamar in earlier versions was that it was focused almost exclusively on the in-guest based source backup model.   While there are cases where this is the right way, there are use cases which are well served with VM-image level backup.   Frustration with those VM-level proxy based mechanisms historically was that VCB had challenges (proxy scaling, increased backup window due to various data movement steps).   All sorts of VM-level variants exist: 1) NFS datastores mounted in other places; 2) Array-integrated snapshots; 3) NDMP backup of NFS datatores; 4) many, many other things.   Each has advantages and disadvantages.

Don’t get me wrong - customers ideally want BOTH in-guest and VM-level, and selectively use them.  In-guest always has the best application-integrated options, broad guest OS file-level recovery and and simple restore/catalog.   VM-level is (in theory minus the VCB challenges) the simplest to deploy/manage, at the expense of move “moving pieces” for file-level restore.

The vStorage APIs for Data Protection in vSphere 4 are focused at improving the VM-level proxy-based approaches.  Avamar 5.0 is out now, and is very much focused on expanding both in-guest and VM-level scenarioes.   There is a TON of vSphere integration.

Key among these:

1. Full support for the vStorage APIs for Data Protection (“son of VCB” – but much better than VCB was).  This means:
   • use of Changed Block Tracking – much better backup performance.    How much faster?  Think 10x-20x faster!
   • direct file-level restore in guests for windows VMs without a guest agent
2. vCenter integration.  This means:
   • Simple views of whether VMs have been backed up or not, and simple ability to remediate.
   • Showing HOW a VM was backed up (guest, VM, not at all) and when.
   • Automatically adding backup policy to virtual machines as they are added.

Beyond VMware-focused features, there are some other massive ones:

• New desktop/laptop backup functionality – a dramatically lightweight client, with simple end-user initiated restore as an option.
• Much more dense Avamar backup target – Avamar has always been awesome for huge reduction of the amount of backup storage needed (30x-50x less is common).   Through a bunch of opimizations, we can get 60% more stuff backed up in the same physical space, for the same dollar cost/watts, etc.   The march of progress!

Networker 7.6 has also been released, and also has piles of additional VMware goodness.   Very good visualization tools that integrate with vCenter, and a much much simpler script-less proxy-based backup model.   It integrates with Avamar, so large enterprises often use Networker for it’s very, very broad application agent and backup device/model support, but leverage Avamar for VMware-centric backup models – but managed from a single console.  Note that until Networker 7.6 gets updated (see below) – the new Avamar functions aren’t exposed in the Networker GUI. 

For customers using Networker on it’s own without the Avamar integration, Networker 7.6 will also be getting similar vStorage API for Data Protection support (for the faster backup gains when using a proxy approach and the single-step file restore from VM-level backup) – but it’s not in the current release.   It is expected in an upcoming near term patch.

If you want to see it in action (remember that this video focuses on what’s new, not the existing stuff), and get the nutshell – watch the below (thank you to Mike Zolla – always awesome in the Avamar product team).  Mike – next time crank up to volume a notch or two :-)

Get the high-resolution version here in MOV format, and here it is in WMV format (slightly cut down, but the same core)

Read on for more detail, as well as a general discussion of backup/recovery in the VMware context!

vSphere update 1 and other friday goodies!

Happy thanksgiving to my American colleagues!

after a bunch of “high level” of posts, glad to do a technical one again :-)

Glad to see vSphere 4 udpate 1 hit today – you can get more here.

A couple important things are improved/added – I’ll call out the key ones for me:

• Windows 7 and Windows 2008 R2 support (32-bit and 64-bit)
• MSCS gets VM HA and DRS support in a limited fashion – you can have MSCS nodes supported (worked before) in a cluster by excluding them from VM HA and DRS.   This means no more “MSCS only VMware clusters”.
• paravirtualized SCSI support for boot disks on W2K3 and W2K8 (nice, eliminates one of the pvSCSI beefs)
• View 4 support (you need update 1 to use View 4)
• This update didn’t trigger re-cert on the HCL, so the stuff that was on the vSphere 4 HCL is there for update 1

A smaller item, but there’s an important fix in update 1 for folks using anything that updated certain things (one of which was any 3rd party Pluggable Storage Architecture SATP, PSP or MPP).   These  they occasionally caused the storage view tab to show the following error (Mike Laverick – you helped us fix this :-) :

The storage service is not initialized. Please try again later.

The following entries might be written to sms.log during the periodic SMS initialization cycle:

2009-04-20 12:18:57,180 [Thread-5] DEBUG com.vmware.vim.sms.provider.VcProviderImpl - Populating scsi volume information...
2009-04-20 12:18:57,289 [Thread-5] ERROR com.vmware.vim.sms.provider.VcProviderImpl - Failed populating service cache
com.ibatis.common.jdbc.exception.NestedSQLException:

Other important items:

1. FCoE/10GbE
   • Got loads of questions from customers deploying gen2 CNAs and wondering when they were on the VMware HCL.   Just checked, and they are there now (see that list here)
   • The Qlogic gen 2 (8100 series) CNA drivers for vSphere are here for FCoE and Network
2. NMP RR (changing the IOoperationlimit value) – important note!
   • we’re getting reports that when customers change the IO Operation Limit from 1000 (default) to 1, on ESX reboot, weirdness happens.   This parameter governs how many IO operations are sent before a new path is selected when using NMP Round Robin.   On reboot if it has been changed from the default to a value of 1 – it becomes what seems a random value – making loadbalancing… less than optimal.   I don’t recommend changing the value until this gets resolved – just leave it at the default.   I expect that this should affect all arrays using block devices equally.   It doesn’t affect PowerPath/VE users– as it is a 3rd party MPP – it replaces the full NMP stack (both the SATP and PSP – and this looks like a PSP thing).  Will do a more detailed post on this early next week.
3. VMware Update Manager (VUM) expanded in update 1 with more PowerPath/VE goodness.
   • Want to see the power of the customer?   I did a survey on Virtualgeek asking “how should we patch 3rd party vmkernel modules?”.   Results?   below:

 

Alright customers – you have spoken!   PowerPath/VE 5.4 can be installed and even patched to PowerPath/VE 5.4.1 (will be released soon, and will of course be the model for subsequent releases) integrated with VUM.  In vSphere 4 udpate 1, VUM gets a “extension” option for 3rd party baselines for vmkernel-level modules and patches (think PowerPath/VE, Cisco Nexus 1000v).   See a demo below.

 

You can download the high-resolution demonstrations in WMV format and MOV format.

FYI – I know that people ideally want PowerPath/VE licensing to be integrated with the new vCenter-centric licensing model – working on it….

Wow – 1TB consumer-grade SSDs hit the street.

Anyone doubting that 2010 will be an inflection point on solid-state storage?   We now have consumer-grade 40GB drive at $115 (talked about here), and a 1TB drive at $3.5K.

http://www.theinquirer.net/inquirer/news/1562474/one-terabyte-ssd-hits-shops

While these are consumer drives, and are missing the things that are needed for enterprise-grade solid state storage that STEC does now (much heaver duty cycle, flat performance over time, larger SRAM to Flash ratios, dual ported interfaces, etc) – it really highlights how fast things are moving, and innovation that occurs in consumer-land can trickle up in the same way that the reverse is true also.

I’m sticking with my bet (admittedly more aggressive than others think) – late 2010 will be the point where solid-state will:

1. match fast magnetic media as measured by $/GB (currently about 10x less efficient)
2. will pass being 300x more efficient measured by $/IOps (currently about 100-200x more efficient)
3. will continue to be 100x more efficient measured by Watts/GB (10,000x more efficient measured by Watts/IOps)

When this happens, will be crazy – only room for SSD (SAS/FC interfaces) and huge slow magnetic media (SATA/SAS interfaces).   Everyone will be looking to see how to maximize this.  

There are second-order effects too – we’re getting close right now to the maximum heat density we can with 15K drives – as we increase the drive density, we can’t use 10/15K RPM SAS/FC, only SATA and SSD (this is why dense – meaning >15 drives per 3U - storage configs generally focus on low-power SATA use cases now).

What people also don’t intrinsically get is that non-volatile storage that sits in between the characteristics of memory, but hundreds (thousands even) times faster than magnetic media has broad transformational characteristics – including things like HPC, BI/DW appliances and all sorts of things.  People have developed entire technology stacks around the idea that you have memory (volatile) and then 1,000,000 times slower you have disk (non-volatile).

Between virtualization (many, MANY more exciting things on that front), broad 10GbE adoption, deduplication showing up everywhere, Nehalem (and it’s successors), and solid-state disk moving mainstream – lots of transformational technology elements – 2010 will be an exciting year!

View 4 – Reference Architectures

Well – it’s finally here!   We’ve been lucky enough to be playing with View 4 here internally for a while – in fact we use it extensively around the super labs as in my earlier post (and for analysts coming to EMC Analyst days this week, you’ll see me using it tomorrow morning…)   We have several hundred View users within EMC, and there is an agressive goal is to make it available to 100% of the employees by mid-year in 2010.

Congrats to Jocelyn and the whole View team – this is a big release, with a lot of additional features, and dramatically improved experience in broader use cases.

As always – loads of work behind the scenes…  You can read what EMC and Cisco have done to support View 4 in the PR here.   So – without further ado, further here are supporting reference architectures and validation work you can leverage…

The pros and cons of being part of Chad’s Army :-)

After last week’s VCE announcement – one unexpected piece of fallout was an influx of great people looking for new exciting opportunities.   Within Acadia, there will be many types of roles.   Within the Solution Support Team, there will be business-development/sales types, “warrior monk” types and “warrior” types.   “Warrior monks” work with many customers, but don’t stick with any one – and this means they travel a lot more.   “Warriors” works with a smaller set of customers stick with them to ensure a positive outcome.  There is no implications of seniority in either model.

We started to jokingly call the role “Warrior monk” and “Warrior” in the early days because it was funny and nerdy in a D&D kind of way, but someone fwded this description and it’s perfect…:

“…the presence, service and dedication of a monk and the absolute skill and precision of a warrior”

Now, I’ll warn people.  There are distinct disadvantages to being on my team.

Here’s one of them – the picture below is some of the folks who are my direct managers (each have their respective teams) in the Americas – we were getting together to plan 2010 the night before the VMUG.   A lot of nights, it seems like we’re the last ones in the parking lot.    It takes passion, commitment, and we do it not because there is someone saying “do this”, but because it just needs to be done.   It can make for a difficult work/life balance (not a good thing) – look I’m emailing this @ 4:30am EST.   The other thing that makes it hard is that we’re a bit “outside” the normal organizational lines.  This sounds great (and is needed to maintain the very fast changes and communication needed), but in practice, extra effort is needed to align with the organizations they support.

On the other hand, there are some upside benefits to being on my team.  This is the Altanta lab – we have 6 of these super-labs (and many more smaller ones) around the Americas, and several in europe and APJ as well.   The goal here is to have all the latest stuff GA stuff, have pre-release everything of VMware/EMC/Cisco – and get real stick time.

 

This is a Vblock Type 1, and also a lot of additional rack-mount and blade servers.     It hosts VM appliance versions of all of EMC Ionix software, most of EMC’s other software (and heck, some of our “hardware” platforms – this lab has many Celerra VSAs, and several of the UCS blades host virtualized Recoverpoint appliances, which really benefit from Palo’s virtual adapter and depends on vSphere’s VMdirectpath).   

It’s constantly in use, and a great source of ongoing learning for the whole team (including our local Atlanta Cisco and VMware brothers/sisters).

We have a ton of both Gen-2 Qlogic and Emulex CNAs filling those Dell 1U rackmounts.   Everything is end-to-end 10GbE (with 4 and 8GBps FC as well if we need it).   Lots of the 10GbE stuff we showed at VMworld 2009 was done here.   Notice the black cabling from the blades and the UCSes (10GbE Twinax), we also are using 10GbE over Cat6a.

  

Everything has it’s price – the team is filled with “go anywhere, anytime, and get anything needed done” attitude (which I love), and get to play with all sorts of the latest software and hardware toys.   But the price for that is a LOT of hard work. 

The passion and love for what you do is the only thing that makes the pace sustainable.

Help a member of Chad’s Army who needs help

I generally want to keep my blog separate from requests for support – but a person on my team has a son with Cerebral Palsy and is trying to raise the funds needed for much needed stem-cell treatment that may have a big impact.   A little donation can go a long way – and think of what it could mean for the Guyadeen family.

Thank you!

http://sites.google.com/site/fundraisingforjustinguyadeen/

Atlanta VMUG

Was a great VMUG in Atlanta – about 700-800 people, and was great to see so many friends, customers and colleagues there!

Cisco did the morning keynote and I did the one right after lunch.   One of my team mates recorded the session, so if you want to see it, you can watch the video below.

Topics discussed included the VCE announcement (near the latter half), the strategy that’s guiding our investments and focus, what customers are telling me, and what we’re trying to do to help them/learn from them…

Here’s the presentation I gave.

Virtual Compute Environment – Is VMware still independent?

This is one of a 5 post series on the VCE coalition announcement, what’s in it, and what it means.  You can see all the posts together here.

Ok – as we sat on telepresence after telepresence, and f2f meeting after meeting approaching the VCE launch, one thing we’re very cognizant of was the fact that some would (and this is a natural fear) that this would signal less independence on VMware’s part.

In uncharacteristic fashion for me – let’s make this short and sweet.   VMware is, and will continue to be, an independent company, that makes it’s own independent decisions, and openly partners with those that partner with them.

if you want more details (and some of the behind the scenes of the VCE announcement with this topic in mind), read on…

Virtual Compute Environment – an insider’s take.

Well – it’s a big day today :-) 

For context – here’s the announcement, and I will also link to the recording of the 3 CEOs discussing what it is, what it isn’t, and what it means.   Here’s also a summary video from John, Joe and Paul.

I’m also glad finally to be able to start talking openly – you should have seen the edits that occured to the VMworld 2009 VMware/Cisco/EMC supersession (SS5240 – which you can watch here) to tiptoe around this (if you do watch it now knowing what we’ve been working on – it’s interesting).   Specifically, the inital versions of that VMworld session called out what we were doing together around Joint Engineering, Joint Solutions, and Joint Support models.   Suffice it to say it go kiboshed by the secret police at all three companies – but we still managed to sneak in a lot of sneak peaks :-)

IF YOU ARE a VMware/Cisco/EMC Employee (and on your intranets) – you can get more content from http://www.vceportal.com

IF YOU ARE part of the public and interested in what we’re doing on this, there is more content here http://privatecloud.com

So, now that the cat’s out of the bag – what exactly are we talking about?

The most important thing isn’t the tech (though that is very cool, and I’ll get to that in a second), or the joint Cisco/EMC/Intel venture (though everyone always focus on that because it is evidence of joint commitment) – it’s an idea.

That idea?   It’s what people who have heard people from VMware (like Paul Maritz or Steve Herrod), or Cisco (like Padmesree Warrior or Ed Bugnion) or EMC (like Joe Tucci or Chuck Hollis or myself) talk in the last year will seem very familiar:

• If you agree that x86 wins (price and price/performance – today mostly, and tomorrow almost entirely), and virtualization is the way to go – it’s possible to build infrastructure based on those assumptions in a much more standardized way, with much more cost-effective and innovative elements, and importantly - leverage VMware and VM-Aware networking, compute, and storage resources to “get out of the plumbing business”.
• This could not only saves a ton of capex dollars, just as importantly, one could get order of magnitude improvements in operational efficiency – instant self-provisioning, end-user chargeback.   This could result in interesting new business and economic proposals, where you could pay-as-you-go as opposed to up front-capex centric models – if you had vendors willing to partner with you in that way.
• It requires infrastructure not only integrated with VMware, but also designed for scale-out, so you can start small, and get as big as you want (otherwise the economic models are whacky).  Also – if you took the engineering resources of industry leaders and applied them in a very focused way against this use case – you could build something so integrated, it would in effect be a software mainframe built out of commodity elements.
• It also requires management that is deeply integrated across the stack – from the infrastructure up to the business application, and that spans the physical and virtual infrastructure.
• Heck – if you could do all that – you’d have standardized building blocks for people building internal clouds, and external clouds (either public or private)

That’s what we jointly announced today – execution against that idea – making the idea real.

To recap – there were 4 major parts of today’s announcement.  I’m going to break this up into 4 posts – one that deals with each.

1. Technology Innovations - Vblock Infrastructure Packages.  Tightly integrated standardized “building blocks”  Click HERE for more on this topic.
2. Integrated Pre-Sales, Services and Support - Vblock Unified Customer Engagement.   Engage like we’re one company, get services like we’re one company, support that is exactly like we are one company.  Click HERE for more on this topic.
3. Solutions Venture and Investment - Acadia. A Cisco-EMC (and Intel) joint venture to build, operate, and transfer Vblock infrastructure to organizations who want to accelerate their journey  Click HERE for more on this topic.
4. Partner Ecosystem Leverage - Vblock Partner Ecosystem.  Click HERE for more on this topic.

There’s huge amounts of detail on each of these things – enough for twenty blog posts (more I bet!).  We’ve been working on this for more than a year (we’ve been plowing away with “intense partnering” for many years, one year ago we got the “engage in joint partnership like nothing else” direction). 

I’m also going to write another one where I will try to answer a question I think we will get often: Where does VMware fit in the VCE Coalition – are they still independent?  The short answer is: ABSOLUTELY.   Go to the link for a more detailed dialog.

It’s an exciting day – and I’m looking forward to having a dialog with you about it!

Virtual Compute Environment – Technology Innovations

This is one of a 5 post series on the VCE coalition announcement, what’s in it, and what it means.  You can see all the posts together here.

Ok, knowing my beloved and technophile readership, you will tend to have come to this link, and not the others (that discuss the other parts of the announcement: integrated pre-sales/services/support; Solutions Venture and Investment; Vblock partner ecosystem).   I’d highly encourage you to look at them, as the business elements are as important as the technical pieces.

Then again, we can’t deny who we are, and I am a nerd at heart that does business too :-)

So – without further ado, let’s discuss the technology innovations!   Read on – this one is long so to spare RSS readers, I’ve broken the post up…