Stephen Foskett's Storage Technology Blog - All Comments

re: What's the Difference Between Compression, Deduplication, and Single-Instance Storage?

Storageologist — Thu, 15 Nov 2012 18:39:10 GMT

There are a couple of other factors to consider in this as well.

1) Nodupe - A pointer based storage architecture uses similar techniques to deduplication to avoid duplicating data in the first place. For instance as snapshots are created they are simply pointer to the original data and space is only consumed as new data is written. This also can apply to the cloning of virtual machines. No need to deduplicate them if you never duplicated the data in the first place.

2) Hardware brute force beats software complexity. Deduplication is discussed extensively as a way to make SSD more efficient from a $/GB standpoint. The problem is not all data can be deduplicated and your mileage will vary. However if you can use 3TB 7200RPM drives instead of 600GB 15K drives you are essentially getting 5x the capacity for the same cost and it will be effectively savings for all of your data if it is compressible, deduplicable or not. Of course this involves innovation in other areas like drive reduild times and effective caching architectures. These exist today though in hybrid systems and it is hard not to bet on the simplicity of simply giving someone more space over unpredictable space saving technologies like compression and deduplication.

None of these techniques are mutually exclusive of course. there are plenty of ways to skin a cat.

Why Are VMware and Storage Companies So Interested In Integration?

Edge2012 Content Hub — Tue, 29 May 2012 18:43:23 GMT

IBM Edge is fast approaching, so I thought it might be nice to give a bit of the sneak preview of my

Why Are VMware and Storage Companies So Interested In Integration?

Edge2012 Content Hub — Tue, 29 May 2012 18:43:23 GMT

IBM Edge is fast approaching, so I thought it might be nice to give a bit of the sneak preview of my

re: What's the Difference Between Compression, Deduplication, and Single-Instance Storage?

Obdurodon — Wed, 02 May 2012 21:24:48 GMT

In my part of the world, people seem to make a slightly different set of distinctions based on two separate axes - granularity and time. On the granularity axis you have file level, block level, and chunk level (i.e. finding similar sections that don't start/end at block boundaries). On the time axis you have inline (in the main I/O path, near-line (while the system is live but not in the main I/O path, e.g. after a file is closed) and offline (while system is quiesced or applied to snapshots/copies). What you describe as single-instance storage seems to be file level near-line dedup in this taxonomy, but I've read papers about "single instance storage" that cover all combinations of granularity and time. Content-addressable storage is a particularly interesting form of block or (less often) file level inline dedup, but they're all separate from compression.