Fixing eagerzeroedthick use - In VI3.5, vSphere and using Zero Reclaim.

As discussed here, in VI3.x, any VMs deployed on VMFS via “clone” or “deploy from template” by default use the “eagerzeroedthick” format.  This was the case all the way up to update 5, which fixed this behavior.   vSphere 4 “clone” and “deploy from template” operations have always used the zeroedthick format (exception is a VM configured for Fault Tolerance, or where the administrator forces the eagerzeroedthick option – for a MSCS/WSFC device for example).

This is important for the reasons covered in that post (check it out) – but the key is that it means you use more storage (a lot more) than needed.

I’ll add another reason – it literally doesn’t just consume more storage, but it makes takes that do this “zeroing” (clone, template) take a lot longer than they need to.

Long and short, even if you do thin provisioning at the array level, eagerzeroedthick VMDKs “cancel it out” – because they “pre-zero” every portion of the VMDK.    Note that production storage dedupe or compression techniques can solve this (since all the zeros can be eliminated), but thin-provisioning at the array level on it’s own cannot.

So, how can you fix it?

• One option to resolve this is to upgrade to vSphere and use the new Storage VMotion.  If you storage vmotion an eagerzeroedthick VM and select “thin” or “thick” you end up with a “thin” or “zeroedthick” VMDK in the end, both of which are more storage efficient.   Also, starting in vSphere, all clone, template operations always default to “zeroed thick”
• Some arrays can do a “zero reclaim”.   This takes virtually provisioned storage devices and “frees up” these “all zero” extents.  This is one of the nice (and free!) little tidbit introduced with the latest Symmetrix Enginuity code (applies to DMX and V-Max) is a “zero reclaim”.   BTW – if you want more on this, you can read up at Barry’s site here.  This looks for any extent in a virtual (thin) pool that is entirely zeros, and frees up that extent for other use in the virtual (thin) pool.   Note that this feature is OS-agnostic.   It will free up “all zero” extents regardless of the host type.

Here’s a quick demonstration of the Space Reclamation (aka Zero Reclaim) as applied in the VMware use case, and also showing how to configure it in the Symmetrix Management Console (SMC).

You can download the high-resolution WMW here.

• If you’re on VI3.5, you can also fix it – independent of array type.    BTW – kudos to the EMC and VMware engineering teams here.   It would have been too easy to just say “upgrade to vSphere 4” or “make sure your array has a zero reclaim”.   The EMC team pushed hard based on customer demand, and the VMware team delivered.    You need to do some specific things to make it work: a) be running ESX 3.5 U5 (Build 207095); b) be running Virtual Center 2.5 U6 (Build 214388); c) configure the ESX host as noted in this VMware KB article: http://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=1017666.   Here are the release notes for U6 with a little more info: http://www.vmware.com/support/vi3/doc/vi3_vc25u6_rel_notes.html#resolvedissues

We’ve seen many customers do either of these approaches on their on their VMware environment and reclaim a LOT of space (40-50%) – and that’s a GOOD THING.

Comments

[Duncan]

If you do a Storage VMotion to go back to a thin vmdk make sure you zero out the VMDK before you do the Storage VMotion. The OS will not zero out when a file is deleted. Only the pointer is removed and thus a SVMotion might not lead to the expected results.

http://www.yellow-bricks.com/2009/07/31/storage-vmotion-and-moving-to-a-thin-provisioned-disk/

[Uddhav Regmi]

No sound......

[Derek]

3PAR can also do zero-page detection and reclimation.

[Chad Sakac]

@Derek - I was careful to not (and in general try to avoid) claims of "uniqueness" - these tend to be very transient.

I would expect that all thin provisioning implementations will get this over time. I know that HDS also has some zero reclaim along with 3PAR.

I'm sure there are variations (perhaps material) but I'm not qualified to judge.

The point of this post was to point out that there are several ways to skin this cat (2 at the ESX host level, 1 at the array level).

Thanks for the comment!

[Uddhav Regmi]

No Sound, please fix that

[Fred Lherault]

Just for the sake of exactitude, what the 3PAR InServ (yes I work for 3PAR) does is not zero page reclamation but online zero detection as they get written by the host, in silicon.

These zeroes are simply not being written to disk so there is no bloating of the LUN in the first place (as would happen without this zero detection during a clone/svmotion on 3.5 or by using eager zeroed thick vmdks)

You dont have to do any manual space reclamation... (although you can if you deleted files and want to reclaim the capacity they occupied)

[Jack MeHoffer]

Chad,

What about the performance gains from using eager-zero thick disks?

Page 6:
http://www.vmware.com/pdf/vsp_4_thinprov_perf.pdf

I am getting ready force the use of eager-zero disks when we upgrade to vSphere for the performance gains. We currently do not have thin provisioning enabled on our arrays but will be doing so in the near future. Will the zero byte reclaim not yield the same result?

In my environment, we delete, create, load, and reload servers often, especially in the test environment. I have timed both operations and these are the results.

Deploy Windows 2003 server:
Zeroed Thick: 5 seconds
Eager Zeroed Thick: 2 minutes 43 seconds

Load time of Windows 2003 server:
Zeroed Thick: 4 hours 40 minutes
Eager Zeroed Thick: 4 hours 19 minutes

As you can see, using eager-zeroed disks allows the server to be loaded about 17 minutes faster from start to finish.

I am not a storage guru so let me know if there are other factors I am not aware of.

[Chad Sakac]

@Fred - thanks for the input, and I'm sure as the early innovator around array-based thin provisioning, 3PAR has excellence and differentiation in this particular area. Of course, I would argue that this is one of many things people look for in their storage arrays. I would also argue that it's an area where the gap is closing (wouldn't say closed). I'd also note that in vSphere 4, inline zero reclaim is offset by use of zeroedthick. Nevertheless, noted - and my respects for a strong competitor.

@Jack - there is indeed a slight performance benefit of the "pre-zero" that you get with eagerzeroedthick. The reason for this is that the vmkernel writes out a zero to that block prior to the actual data IO. IMO, except for the most extreme circumstances, the upside is outweighed by the downside - and that zeroedthick (or thin) is the way to go. Furthermore, in the next vSphere release, there is a vStorage API (engineering name is "write same/zero") that "hardware offloads" all this "zeroing" activity, which will close that gap even further - **IF** you are using a VAAI (vStorage APIs for Array Integration) compliant array (current generation EMC platforms will have this support with a software update). The array can do that more efficiently than at the ESX layer as there is the ability to batch up the IO operations en masse.

[orz]

I actually do create my templates as eagerzeroedthick disks, since the performance gains (http://www.vmware.com/pdf/vsp_4_thinprov_perf.pdf) are at not costs/drawbacks for us.
We don't use SAN thin provisioning or anything ``fancy'' like that.

The clone operations are still pretty fast too.