Has the Time Finally Come for Data Reduction?

In an age when compression and optimization of data pervades the lives of consumers, it seems odd that enterprise IT has been slow to adopt the same technology. Compression enables portable music players, digital mobile phones, and HDTV, yet we still store raw “fat” bits on our storage arrays. It’s time for enterprise buyers to catch up with the consumer electronics world!

Imagine if your iPod only held 10 CDs worth of music. Although still somewhat useful, such a device would not be a compelling purchase. But add in MP3 compression in the same device holds 100 CDs and becomes a must-have gadget. Lossy and lossless data optimization technologies make it practical to watch movies on a plane and bring high-definition video and homes. Without compression, media would be stuck in the 1970's!

Enterprise IT is just opening its eyes to the possibilities of compression and data optimization, even though these technologies have been around for a long time. Computer users began employing compression and archiving almost immediately after personal computers were released. I myself employed Stacker to double the capacity of the 20 MB hard drive in my PC, and gzip is a staple on my UNIX machines.

But enterprise storage, on the whole, does not use any sort of compression technology. Enterprise IT relies on full, “fat” bit streams from the SAN to the disk drive. We are willing to spend millions of dollars literally storing nothing, and enterprise storage vendors are perfectly happy to cash those checks!

Initially, it was felt that compression and data reduction would overly impact performance on enterprise storage systems. Indeed, whole scale compression is somewhat CPU intensive. But today's storage systems benefit from the same vast improvements in processing power as desktops. Moore's law has brought compression within reach, if only customers would adopt it.

Thin provisioning has become much more common in enterprise storage systems in the last few years. This can be seen as a sort of first step toward wider acceptance of data optimization technologies, since thinly provisioned storage breaks the strict 1:1 relationship between allocated and usable storage capacity.

Many buyers remain leery of thin provisioning, however, fearing that users will overflow actual capacity and bring systems crashing down. Software vendors like VMware have done much to mitigate these fears, incorporating reporting and alerting features. It is likely that future operating systems will become smarter about alerting, and will follow VMware's lead in incorporating smart “stun” features to mitigate the impact when capacity is truly exhausted.

De-duplication has gained some acceptance, as well, especially for nonproduction data. It would simply not be practical to store backups and archives on disk without de-duplication technology, and most enterprise storage vendors have a solution for these needs. The next step will be incorporating de-duplication technology in primary storage devices, a move that some vendors have already made. Again, however, IT administrators are moving cautiously when it comes to primary de-dupe.

De-duplication of data can be thought of as a special case of data compression, and many are working on improving the efficiency of data storage beyond today's products. Indeed, it is likely that future data optimization technologies will be effective enough to bring the cost of flash memory within the reach of mainstream IT shops. Smaller vendors are already promising this, and we will likely see this combination of data optimization and flash spread throughout the industry in the coming years.

The core issue is not in the technology of data optimization but in the comfort level of enterprise IT buyers. Some audiophiles don't like MP3s, but even they are willing to adopt lossless compression schemes like FLAC, the musical equivalent of thin provisioning, since it enables greater flexibility and portability. It is likely that the benefits of data reduction and optimization technologies will similarly tempt enterprise storage buyers in the future. Then enterprise storage will finally catch up to home entertainment!