« DRS For Storage! | Main | At VMworld? Try VPLEX. Like it? Take one home :-) »

August 31, 2010

Comments

Feed You can follow this conversation by subscribing to the comment feed for this post.

StefanJagger

David Flynn from Fusion-io spoke about a similar concept just now on theCube/live@VMware which I commented earlier.

Umesh Maheshwari

Chad, thanks for writing about this very interesting issue about how best to use flash within a storage system.

As I understand it, you are making these points:

1. Using flash as a tier with auto-tiering is helpful in many cases. However, it is relatively slow to respond to changing workloads. Therefore, it is helpful to have a complementary cache.

2. Using flash as mega cache is helpful in many cases. However, it has the following two limitations, so it is helpful to have complementary tiering.

(A) The cache takes time to warm up to hold the working set.

(B) The cache is smaller than a tier could be, limited by amount of memory needed to index it.

I agree with point 1. Besides being relatively slow to respond, tiering often works at a coarser granularity than caching. E.g., the unit of tiering might be 1MB, while the unit of caching might be smaller than 100KB. This might be because tiering is heavier weight than caching.

I am not sure about the limitations in point 2.

Regarding limitation (A): If the data could be in a fast tier to begin with, because of either manual or automated assignment, the data could as well be in the cache, because it could be tagged as being highly cache-worthy either manually or automatically. In particular, it could always stay in cache---starting from the very first writes to each block, so that there are no cache misses.

Perhaps you are suggesting that, if the cache has to hold a large fraction of the dataset to provide a significant payback, it is better to just put the dataset in a faster tier and be done with it. Let's consider the pros and cons of having all of the data in a flash tier vs. having all of the data in a flash cache.

- The advantage of putting the data in a flash tier would be that it would not use any disk space or disk write IO. (Neither approach will use any disk read IO because each would serve data from flash.)

- The advantage of putting it in a cache would be that the cache can tolerate data loss and therefore does not require expensive flash with high endurance, parity or mirroring, or a whole bunch of other mechanisms needed to ensure durability.

In most cases, the second advantage outweighs the first. This is because the dollar savings in disk space are small because disk is much cheaper than flash. Furthermore, flash as a media is actually worse on writes than disk. The reason some systems use flash for "write buffering" is
that the flash SSD vendors have implemented write coalescing in the SSD firmware. One could implement this write coalescing on top of disk and get similar benefits without using flash. This is the approach we have taken at Nimble Storage; see my blog at http://bit.ly/dqHymr .

Re limitation (B): it is true that treating flash cache as an extension of DRAM cache would limit the size of the flash cache. However, it does not have to be so. If one implements the flash cache like one implements on-disk datastructures---using a hierarchical tree of blocks---one can
build a scalable flash cache.

Nonetheless, on the whole, I agree with your general message that there are many different ways of using flash to support applications. E.g., flash can be used on the server as well as within storage.

Umesh
CTO, Nimble Storage

The comments to this entry are closed.

  • BlogWithIntegrity.com

Disclaimer

  • The opinions expressed here are my personal opinions. Content published here is not read or approved in advance by Dell Technologies and does not necessarily reflect the views and opinions of Dell Technologies or any part of Dell Technologies. This is my blog, it is not an Dell Technologies blog.