« Psst- want to see some UCS systems in action? :-) | Main | Oracle on VMware a manifesto »

April 30, 2009


Feed You can follow this conversation by subscribing to the comment feed for this post.

the storage anarchist

That latter "exception" doesn't really apply to an intelligently cached disk array like the Symmetrix DMX and V-Max.

The bias towards "IO Locality" really only works for locally-managed storage - with external storage, the actual locality is rarely as expected, since the array will distribute data across a large number of spindles or pack it into cache such that the "locality" no longer applies. Since databases rarely do long-run sequential reads (where minimized head movement might be of benefit), for the Symmetrix implementation of Virtual Provisioning you are probably still better off using the array-based thin implementation than the host-based approach.

YMMV on other platforms, of course.


While I completely agree with anarchist's premise that IO locality is not a material issue for an intelligently cached disk array, I am surprised by the unwritten message that only uber-expensive high-end arrays alleviate this issue. Only Symmetrix DMX and V-Max offer this? I can think of several mid-range priced arrays that are leaders in array based thin provisioning (and have brilliant caching and data layout schema) that perform extremely well in VMware environments.

YMMV is right and it may be better ;-)


My previous post was a little quiz. To make it a little easier for ya, here's a disclaimer I should have put on the trailer of my comment: [I work at 3PAR] :-)

Chad Sakac

Thanks for the comments Rob, Anarchist. I'll update the article accordingly - I think you are right on the ONE SMALL point in the whole article - the curse of the blog is that people tear apart one paragraph in a full post :-)

Rob - I'm missing 3Par on the vStorage API calls on the TP mgmt integration as "leaders in array-based thin provisioning" you guys would be leading the charge :-)

3Par makes a fine product, and kudos for being one of the earliest pure block devices to ship TP. That said - it is now relatively mainstream - and Anarchist's comment doesn't have a hidden unwritten agenda - he's a plain-jane, up-front Symmetrix bigot (it's the part of the company he works for after all) - as I am a VMware bigot (though try to be as pragmatic as I can).

As he himself said - IO locality is only really needed in local storage use cases. The same comment would apply to all EMC arrays, mid-range and down. Every array varies, and customers should evaluate carefully on all fronts.

The guidance (use array-level TP, Thick on Thin and Thin on Thin yield roughly the same efficiency, only use Thin on Thin if you are carefuly about thresholds/alerts/actions) holds.

Paul Wegiel

Great insightful article Chad, thanks a bunch!


Thanks Chad - excellent article nonetheless. Very well done!


Dejan Ilic

Actualy, there is a third angle on the "thin" story.

We have had the problem with clones inflating and taking a lot of space. Also, due to some conservatism among sysadmins everyone claimed they needed 36GB or more on the servers so we had to give them the space.

It didn't help that we are using Vmware on NFS.

Then we implemented deduplication on the primary storage on our Netapp. So now I have the best of all the worlds, both inflated Vmware disks and yet using minimal space as all the zeroed blocks (and MORE) is deduplicated away, totaly transparently to Vmware and the Vm-systems sysadmins.

Sure, thin-provisioning the Vmdisks, in Vmware or "classical" array-thinprovisioning would probably releave some pressure from the dedup stage, but I have no headache anymore with sysadmins choosing the wrong format. And deduplication also releases more storage as it finds identical OS-related blocks, something that Vmware thin provisioning isn't able to get away with.

Chad Sakac

Dejan - thanks for the response.

I'm actually working on a multivendor NFS post with my NetApp colleagues - it's a great option for many use cases - hopefully we'll get it done soon.

Yup - that's also a way to be capacity efficient. I would argue that there's still a benefit (even if using production storage dedupe) to the new thick and thin defaults - cloning/template operations will be much faster, and you can avoid shrinking the flexvol - or over-provisioning before the post-process dedupe (unless using file-level snapshot approaches - which are an option in some of the use cases)

I believe that over time - production dedupe will be a ubitquitous feature - each vendor does certain things first, and NetApp did production dedupe first. We've just introed our first production dedupe (you can download the Celerra VSA and give it a whirl), and will continue to evolve this as fast as we can to broaden use cases and platforms - efficiency needs to be applied at every level, every use case, and every location.

Deduplication techniques in general have broad applicability - backup, production, IO - everywhere.

Again - thanks for the comment!

Paul Wegiel

I am wondering how VMware will handle disk fragmentation... I remember, "growing up" on GSX and Workstation (before ESX was widely popular) - thin provisioning was a feature of both GSX and Workstation and disk fragmentation was always a big issue as time progresses.

Will VMware have some sort of defrag utility that will help defrag the thinly provisioned disks, which now may now end up all over the VMFS?

I have not seen any mention of a built-in defragmenter so far.

Tom Howarth

I think the issues of Defrag will only be apparent on local storage with less spindles and traditional RAID based partitions, the use of "Volumes or Meta LUNs" etc on Remote Shared storage effectively spread the VMFS partition across many more spindles than traditional RAID based partition ever would.

Chad Sakac

I tend to agree with Tom - I've never had this come up as an issue at any customer (which suggests it might be a "mountain of a molehill").

You can't think of workstation and GSX - as they of course live on the host filesystem.

Fragmentation is inherently not as bad with VMFS purely because you have a relatively small number of very large files compared with a "normal" filesystem use case.

VMFS-3 also doesn't "spray it all over the place" - initial file allocation is random, subsequent attempt to be sequential. This means the contents of files are grouped, though the files themselves are all over.

Of course, doing a storage vmotion to a new datastore would also "defrag" it, so there is already a non-disruptive workaround.


I am curious to know if the Windows servers need to have the drives formated in the dynamic or basic format. The physical servers require me to be in basic format now to extend from the SAN but dynamic drives can be extended if I need to grow them. Can I format them in dynamic and grow them in 4.0 thin provisioning or do I need to keep them all basic and forever lose the ability to extend those volumes.

Chad Sakac

@Kevin - you can have them basic and still expand them. You can grow basic disks using diskpart. In W2K8 that gets even easier (it's right in the GUI).

Paul Evans

Excellent post, Chad. I found it from Vaughn Stewart's post and am pleasantly surprised to discover convergence of viewpoints around the fundamentals between your respective posts. I learnt quite a lot from them and have summarized my learning at


Thank you


Is there any way to convert my eagerzeroedthick disks (created form templates on VI3) tot the vSphere default (lazy)zeroedthick format?
I want to use only the thin provisioning on the storage array and not thin on thin. According to vmware support this cannot be done.

Son Cao

You can go from eagerzeroedthick to zeroedthick by performing a Storage vMotion. After selecting your datastore, you have the option of choosing a format: same format as source, thin, or thick. Note that by selecting thick, you are choosing zeroedthick and not eagerzeroedthick.

Terry Tsang

Say a VM running MS Windows was newly created using TP, and a 10GB file was just copied to this VM. The size of the vdisk, as expected, would be expanded by 10GB. How to reclaim this 10GB space after the file is permanently deleted within the VM?

Peter Tak

@Son Cao
Just tested migrating to thick but it does not work like you say, the new disk takes up all allocated space on the virtual provisioned lun.
This is very different from creating a thick disk from scratch (that only takes up the used space in the vm and not all allocated space)
Anyone have an answer?

buy generic viagra

Last time I had some contact with this and really was not very happy with the situation just like me that the selection of virtual disk format is available in the creation of graphical user interface.


First, thanks for another in a long line of great posts.

As this post was written about 10 months ago I thought I would see if anything has changed you opinions on TP, in particular Thin on Thin.

I am currently running a VI 3.5 environment using a CX4-120 w/o TP. I am about to move to vSphere and am also looking at possibly replacing the CX4-120 with a NS-480. I am looking at the possibility of using TP and probably leaning towards trying Thin on Thin.

Along with this, on the NS-480 I am looking at using FAST for tiering with about 1/3 SATA, 2/3 FC and a small amount of SSD. Does the introduction of tiering with FAST change have any impact on the use of TP?



can you please explain the following statement in regards to "Thin on Thin" can have an accelerate "out of space condition". The storage used by the VMware environment is NFS Datastores on a Netapp storage array. We currently have Thin configured on the VMware layer but not yet on the Netapp storage device. We want to turn on Thin on the Netapp but would like to obtain anyone experience and feedback and any issues they have encountered using Thin on Thin with Netapp NFS.

What if your array DOES support Thin, and you are using it that way - is there a downside to “Thin on Thin”? Not really, and technically it can be the most efficient configuration – but only if you monitor usage. The only risk with “thin on thin” is that you can have an accelerated “out of space condition”.

Chad Sakac

@morjo02 - thanks for the comment. I've often observed that people struggle to track a metric in two places (ergo track storage utilization at BOTH the underlying storage AND at the VMware datastore).

This is often compounded when you're talking about two different teams.

Space consumption of Thin on Thin obviously consumes the same (real) resources as Thin on Thick, but enables you to oversubscribe even further. In your case (and this applies across the storage vendors, just using different "words" for features):

- your FlexVols will be oversubscribed (what is shown as 2TB for example to the ESX host, might only actually HAVE 1TB).
- your Datastores will be oversubscribed (It will LOOK like you can put in 20 x 200GB VMDKs for a total of 4TB - even though the datastore LOOKS to be 2TB in size, though of course, it might only have 1TB under the covers).

That double-oversubscription means that as you approach the actual allocation limits, the process of "running out of space" actually takes less time, giving you less time to respond.

If you are committed and able to track & manage at both layers (which isn't rocket science, but falls into "operational excellence") - you're fine with Thin on Thin. My only guidance is understand how important monitoring - via Managed Datastore Reports in vCenter 4.x and your storage array's tools - is going to be to have a very available infrastructure.

David H

Chad, is there any tuning to be done on the Clariion pool side for thin lun performance? I've been messing with different configuration options for my ESXi 4.1 w/powerpath to CX4-480 setup and if I thin provision on the vmware side, thick LUN on the EMC side, I get nearly identical write performance to thick on thick, but read performance is only about 60% of what it is thick on thick. If I instead thin LUN on the EMC side and thick provision on the ESXi side, I gain huge on the read side, getting nearly the same as thick on thick, but my write performance drops to 33% of what it was thick on thick. So I'm stuck with a trade off of vmware thin if I want good writes but poor reads, EMC thin if I want good reads but much slower writes.

I supposed I need to do some playing with the clariion's cache allocations, maybe that would help.

Mark Singer

I appreciate all your fine work here but perhaps you could explain more clearly what benefit we get from Thin on Thin. Let's just assume there are additional scales of economy and performance obtained from TP at the array level.

What is that TP at the VMware level brings me specifically ON TOP of this when I go thin on thin.

Indeed, intuitively I would think it just adds overhead to the hypervisor, potential SCSI reservation contention (after all, don't increases in VMDK size require a SCSI reservation due to meta data updates?). I know you have a reason in mind but I'm not quite picking it up. Any way you might clarify? Thank you again for your post!

Tom Kivlin

Hi Chad,
This is a great article and certainly answers many questions; is the information contained within still relevant in vSphere 5(.1)?

Chris Anania


Eventhough a Think VMDK allocate all the space up front, is it really? Isn't it just a point to the end of the file? If so isn't the stroage array able to determine the real about of blocks being used in a Think VMDK on a Thin LUN using VAAI?

Verify your Comment

Previewing your Comment

This is only a preview. Your comment has not yet been posted.

Your comment could not be posted. Error type:
Your comment has been saved. Comments are moderated and will not appear until approved by the author. Post another comment

The letters and numbers you entered did not match the image. Please try again.

As a final step before posting your comment, enter the letters and numbers you see in the image below. This prevents automated programs from posting comments.

Having trouble reading this image? View an alternate.


Post a comment

Comments are moderated, and will not appear until the author has approved them.

Your Information

(Name and email address are required. Email address will not be displayed with the comment.)

  • BlogWithIntegrity.com


  • The opinions expressed here are my personal opinions. Content published here is not read or approved in advance by Dell Technologies and does not necessarily reflect the views and opinions of Dell Technologies or any part of Dell Technologies. This is my blog, it is not an Dell Technologies blog.