Real World Experiences Using Storage VMotion

One of the most disruptive events for customers is often a storage platform upgrade, since everything is consolidated - ergo everything is potentially affected.   EMC has a multitude of ways to do this... lemme think... right now I can think of 4 (non-disruptive platform upgrade in many CX cases, PowerPath Migration Enabler, Invista/Rainfinity, and OpenMigrator/SANCopy).   Every one has a specific use case (every customer is different).    But  a new tool in the toolbox is making a strong entrance on the migration scene for customers using any VI 3.5 release - Storage VMotion. 

As you'd imagine, EMC does a lot of these data migrations, and recently there was a big internal thread on this I thought it would be good to share, as well as sharing customer experiences.... 

Storage VMotion is obviously only useful when the data is VMware, and (for now) iSCSI and FC only.   It works generally with NFS datastores , it's not supported - I've seen some flaky cases so I don't think VMware is being arbitrary.   NFS as a source and a target is expected to be officially supported in the future.   

That caveat list is pretty short.   If it is VMware data (one more great reason to Virtualize?!), svmotion can be a fantastic option, and joins the other mechanisms as a key tool.

Read on for a couple of my svmotion "rules of thumb" and a couple customer examples....

Ok - under the covers, svmotion:

1. Takes an ESX snapshot
2. Then file copies the closed VDMK (which is the large majority of the VM) to the target datastore
3. Then "reparents" the open (and small) VDMK to the target (synchronizing)
4. Then "deletes" the snapshot (in effect merging the large VDMK with the small reparented VMDK)

Source: VMworld 2007 - TA54 Technology Preview –Storage VMotion

Ok, common questions:

Q: Is Storage VMotion really non-disruptive?

A: Well, if you're a bit of a propeller-head/nit picker (as I'm wont to be) - it's "non-disruptive" (i.e. doesn't require an outage) but can have a "performance impact".   During the vast majority of the operation, the source datastore needs to continue to support the production workload and the additional workload of being the source of the filecopy process.   This filecopy isn't governed or throttled in any way, so it WILL become the heavy IO consumer at the expense of the production IO.   My suggestion: even though it involves NO downtime, still schedule svmotion operations.   They'll complete faster, too!   BTW - this is one of the aspects of the vStorage API set that's being worked on - it's possible to offload the copy/move task from the ESX host on a VM per VM basis, for all types of datastores (VMFS - where it needs to involve a blocklist-type mechanism, and NFS where it doesn't).

Q: Why doesn't EMC and VMware coordinate better - after all EMC has SAN/NAS virtualization, doesn't that compete with Storage Vmotion?

A: If you ask me, Storage VMotion and SAN/NAS virtualization are complementary, not competitive.   First of all, I've said it before, and I'll say it again, I think that vendors who equate storage virtualization (in the classic HDS USP, IBM SVC, EMC Invista/Rainfinity sense) with VMware are making a technical error.   The core value prop of server virtualization is threefold: 1) Consolidation for efficiency; 2) flexibility via non-disruptive mobility; 3) hardware independence via software.   Most customers only have one array per major site - and those customers get 1+2 (but not 3) in any modern array (to varying degress).   SAN/NAS is inherently consolidated and highly utilized relative to DAS (the physical->VM server analogy).   Most modern arrays have the ability to add/move/change almost every attribute non-disruptively.   What SAN/NAS virtualization add is the ability to do that over multiple heterogenous SAN/NAS arrays.  None of these provide "hardware independence via software" (although it's arguable whether any of these things get you independence or simply a transfer of vendor control points).

SAN/NAS virtualization approaches typically have a data mobility rate that's still an order of magnitude faster than a really efficient Storage VMotion.   So - they are adjacent use cases: i) Storage VMotion is a laser (you can Storage Vmotion a single VM, or even part of a VM), conversely Storage Virtualization is a hammer - fast, larger volume, but not as focused; ii) Storage Virtualization might be a better answer if you have many SAN/NAS devices, and they are heterogenous; iii) Storage Virtualization can do it's function with host independence - Storage VMotion is VMware-only (take that Hyper-V)....    Long and short: Storage VMotion is a useful thing for almost everyone, SAN/NAS virtualization is useful for a subset of customers.

Q: How fast is a a Storage VMotion operation?

A: It is primarily governed by the performance of the source and target datastores, but in my personal experience, 0.5GB/min is a good rule of thumb for a decently configured system.

Q: Can I do multiple Storage VMotion operations in parallel to make it work faster?

A: YES!   You can do them both in parallel in the sense of simultaneous operations on the same datastore, as well as in parallel across datastores.  

First -  simultaneous operations on the same datastore.... Regarding If you look at this guide page 246.

"VMware Infrastructure 3 supports a maximum of four simultaneous VMotion or Storage VMotion accesses to a single datastore. A migration with VMotion involves two simultaneous accesses to the datastore, by the source and destination hosts. A migration with Storage VMotion involves one access to the source datastore and one access to the destination datastore. Therefore, if no other migrations are occurring, up to four concurrent Storage VMotion migrations involving the datastore can occur simultaneously."

BUT - in my experience, you don't net a lot from this.    Normally with a single filecopy, there's some benefit of simultaneous multi-threaded copy processes, but remember, it's likely that this datastore already has a bunch of IO that it's doing.   Can it help?   Sometimes - but personally, I'm all for the next thing....

Second - parallel across datastores.... Absolutely leverage this.    If you do it - you can get N jobs at once, where N is  the number of datastores you have on the cluster, but cannot exceed 32 (because you can only have one SVmotion per ESX host, and a max of 32 per cluster with VI 3.5)

Q: I don't like the CLI, I really wish there was a GUI....

A: The CLI isn't that bad, and if you ask me is the right way to do this en masse (we've done engagements were we get a list of VMs from the VC DB and use that to automate the overall svmotion exercise).    BUT - if you want a GUI - I've had OK experiences with this third party VIC plug-in: http://sourceforge.net/projects/vip-svmotion/   Good news is that the next VMware major release will add a formal GUI.

Q: Are you guys working on something in Storage Vmotion using the vStorage API efforts?

A: Yes.   We think that if we have the block-list (or file data) to identify a VM exclusively, we can reduce the performance impact of the Storage VMotion filecopy step dramatically, within an array for all our arrays, and across arrays using Invista for Block and Rainfinity for NAS.   More to come - but work is underway.

A couple customer Examples:

These are recent customer migration projects I know of....

• "We recently (September) did a svmotion migration of 156 VMs comprising 3.2TB of data (dozen luns) from a CX700 to a CX3-80 and finished in 16 hours .. Averaged 200GB/hour (33GB/10mins), and that's with management overhead of manually managing the svmotions (no automation)."
• "They have migrated most of the 600 to one of the new CX3-40s and are only seeing around 50% SP utilization.  That is during backups and simultaneous migrations.  The customer moved a large (40,000 iops oracle box).  It took 7.5 hours to storage vmotion across 1.7TB worth of machine\storage."

Customers, VMware/EMC Partners - what has YOUR svmotion experience been?

Comments

[Duncan]

Keep in mind that the snapshot is placed on the source disk, so you need additional space on he source. the amount of space needed depends on the changes written to disk.

[Chad Sakac]

Thanks Duncan - absolutely. There is a requirement of space on the target, and incremental space on the source (for the period during which the snapshot is diverging as the filecopy of the closed VMDK is occuring).

[Thorsten]

Well, a perfect innovation would be something like a failover scenario using a combination of storage VMotion and the new non-disruptive / fault tolerance component for SAN storages.
E.g. I was (and still am) search for a possibility to have a fault tolerance solution with two MirrorView'ed SAN-LUNs. So if one of the SAN-storages fails ... nothing will happen - the ESX should just continue to proceed with the second node...
If this will be possible on VMFS - I guess this is gonna be the killer-app :-)
(no, DataCore/LeftHand is not an option as it complicates a structure again with an additional abstraction level / tool / hardware piece )

[Roberto]

Could you say something more about storage motion using rdm ?

I have the most of the server in simple vmdk. Just 4 of them use rdm lun, the biggest lun I have are rdm (file server and email server)

thanks

Roberto

[KrzyWi]

I echo Roberto's request, what is your experience with migrating rdm (both virtual and physical compatibility mode) from one SAN to another?
I will soon have to do this for one of the customers who has got quite a few MSCS clusters and would like to find out more about real life experience and best approaches.
I know you can use "vmkfstools -i" to do this, but like I say going through successful studies would be of great benefit!

Thanks
Kris

[Vikash]

Say if you are doing svmotion on same filer on the aggr/vol or different aggr/vol. When we kick this does it go and write via network ? or does it copy from one aggr/vol to different aggr/vol on the filer ?

Thanks,
Vikash

[costa rica hotels]

I really liked your page is very good and gives me a lot of information is the only web site and seen that it is complete across the network and searched and bucado and could not find another like me not in accordance with information from other Web sites look for one with more information and found it very complete congratulate