Stephen Foskett's Storage Technology Blog

What's the Difference Between Compression, Deduplication, and Single-Instance Storage?

2012-05-02T19:41:00Z

When I talk about the various state-of-the-art capacity optimization solutions that are now appearing in the market, the same comment usually arises: "Isn't this the same as zip?" Or a long-tenured storage pro will remind me that, "Stacker died out a long time ago, so why is this any different?"

These are good points: The difference between traditional compression and modern data deduplication is somewhat hazy. And it doesn't help that various implementations fall all along the spectrum from "mildly interesting" to "cutting edge!"

Over 100 years ago, Samuel Morse defined a coding scheme for text messages. He optimized the efficiency of transmission by using fewer bits for common letters ("e" and "t" are a single dot or dash, respectively) and more for uncommon ones ("w" is a dot and two dashes). Morse Code remains in active use to this day, and the concept behind it lays the groundwork for binary transmission technology used by computer systems today.

Pure data compression solutions apply Morse's idea to a general set of data, with the core idea being that longer sequences of bits can be represented by shorter ones. Huffman Coding, a general mathematical mechanism to encode a string, allows a simple sequence of bits to represent a letter or part of an image. The compression engine replaces a longer sequence with a short tag that tells the de-compressor to substitute the correct, original sequence.

Other data compression techniques exist as well. DVD and MP3 use lossy compression engines that "throw away" data that is deemed unnecessary. The resulting output is not identical to the original content, but TV viewers or music listeners might not notice the difference.

Simple compression technology is easy to implement but limited to relatively short data sets. This makes pure compression useful but limited.

Single-instance storage (SIS) is another simple compression concept. Rather than letters or bit sequences, single-instancing looks for identical files or objects. If two people saved the exact same file, such a storage device would just save one copy of it, maintaining a pointer for consistency. Single-instance storage was a key component of Novell GroupWise in the 1990's, and was part of Microsoft Exchange until recently.

Single-instancing is simple to implement but limited in effectiveness. Although duplicate files occur with reasonable frequency, they aren't nearly as common as files that differ only slightly, including revisions of a similar document or presentation. This technology has had a minor resurgence as part of cloud storage services but is fairly uncommon in the enterprise today.

Data deduplication introduces a novel twist on single-instancing: Rather than looking for entire duplicate files, deduplication engines search for identical blocks within a file or data set. This makes "dedupe" much more effective in practice, since such systems may catch slightly-different files. But it is also much more difficult to implement such systems effectively, since there is significantly more data to process.

Since there is no perfect universal block size, some deduplication systems use variable-sized blocks. They evaluate a data set and determine what size to use based on best practices, similarities to other data sets, or trial and error. Some will also "unpack" bundled objects, looking for embedded media and the like.

Deduplication is so computationally intensive that many systems will postpone processing until after-hours or when the system is idle. These "post-processing" systems store all incoming data in raw form, only "sweeping" through the data later. This isn't as bad as it sounds: These systems usually perform just like native storage and won't waste energy deduplicating transient data.

Data deduplication has proven effective in real-world enterprise storage and backup systems, and variations on this technology are becoming widespread. But each system is somewhat different in effectiveness and performance, so buyers are wise to try them out before committing to one or another. And many mix in compression and thin provisioning, too, making the resulting device even more efficient.

Why We Won't See the End of Hard Disk Drives Any Time Soon

2012-04-23T19:24:00Z

With solid-state storage getting all the attention these days, it's easy to think that "spinning rust" is headed to rapid obsolescence. SSDs are an order of magnitude faster, quieter, and more energy efficient. Why should we use heads and platters when chips to it so well?

A similar line of reasoning has haunted tape for a few decades now. Open systems never used tape for primary storage, and disk-based backup and archiving are the darling of storage array vendors. Yet just like so many technologies, it's hard to overlook just how great tape is for streaming performance, off-line storage, portability, and media price.

Like tape, hard disk drives are very good at certain things. Spinning disks strike a nice balance between random- and sequential-access performance, and nearly every current operating system and application was designed with this profile in mind. Although flash memory absolutely slaughters disk when it comes to random reads, its lead on writes isn't quite as impressive.
Tiered storage and disk caching are also a factor. Numerous companies have proven that a small amount of RAM or flash memory can dramatically accelerate storage system performance. Nearly every enterprise array now includes automated storage tiering or caching, and this blunts the performance benefits of solid state storage.

Then there is the financial angle. On a capacity basis, hard disk drives remain a tenth the cost of finished flash SSD drives and boast far-greater total capacity to boot. Buyers looking for massive capacity find flash memory quickly priced out of contention unless data can be compressed and access is thin-provisioned.

But there is an even more important fact to consider when it comes to the alleged death of hard disk drives: There simply isn't enough flash memory on the planet to eliminate spinning disks! The annual consumption of hard disk capacity dwarfs the capacity of flash memory foundries, and ramping up production is inconceivable.

Why not just build more flash memory factories? For starters, flash is built from silicon wafers, which much come from another factory. And these are already in demand for other solid state components, including CPUs and RAM chips. It would take years to reach the levels of capacity of today's hard disk industry, and few companies are willing to invest in flash memory production.

Critically, though, producers are scared off by another looming issue specific to NAND flash, the current solid state storage media of choice. As NAND cells get smaller and more dense, they lose their ability to be reused. Already, 2x nm NAND flash is an order of magnitude less reusable than the previous generation, requiring ultra-smart controllers to manage the chips.

It's not clear that NAND flash has much life in it, so the solid state storage industry is looking to other technologies, including ReRAM or so-called memristors. But these are brand-new technologies, and it's not clear which, if any, will replace NAND. This uncertainty makes investment even more risky for producers.

Current flash memory technology simply can't knock off spinning disks, and the future is uncertain. Simply put, it will be a decade or more before the last hard disk drive is produced.

The Working Set and the Cloud

2012-03-28T00:45:00Z

We storage people like to talk about primary storage, but what exactly does that term mean? The conventional definition would include both up-to-the-minute transactions and decade old files, as long as both are available for immediate access. Yet it is clear that these are different use cases, and today's discussions of tiering, cloud storage and "big data" demonstrate a need to treat them as such.

It's time to reconsider the old definitions of primary, backup, and archive data. Rather than considering all online data to be "primary", why not instead ask ourselves what data is needed and when. Current applications and processes require a "working set" of data, and access to other types can wait. Thinking of data in this way allows us more flexibility when locating data and prioritizing protection.

The working set is not entirely defined by access patterns, but this can be a valuable clue. Recently-accessed data is much more likely to be critical to near-future operations, and files that have been untouched for years are unlikely to be needed immediately. This is the fundamental mechanism used by most automatic tiered storage devices, from caching to cloud gateways, and it works reasonably well. But applications can do better.

What if an application tagged data with an index of its "staleness"? We could relocate stale data into the cloud, yet still keep it accessible if needed. More importantly, we could prioritize "hot" data, placing it on faster and closer storage, or even caching it in the server. We could also make it a priority for recovery of in the event of an outage or disaster.

The advent of cloud storage means we had better consider this question, and soon. The volume of data being created in and moved to the cloud means we must all begin focusing on the working set.

Data has inertia. It takes a finite amount of time to read or write data. Although this varies based on the storage media used (e.g. flash is faster than disk or tape), the volume of data makes a much-larger difference. It takes quite a bit of engineering and expense to move a terabyte of data in an hour, regardless of the storage media used. Ever-larger volumes of data similarly requires more-substantial expense.

There comes a time when data can no longer practically be moved. Once this point has been reached, an extensive outage or migration is the only way to remedy it. Data becomes "trapped" by the slow interfaces used.

But why move data at all? What if we simply declare victory, storing as much data as possible in the cloud and focusing instead on optimizing the identification and acceleration of the working set? Suddenly, things change.

If computer scientists and application developers focused on the challenge of identifying the working set and making it available, cloud storage becomes far more useful. We can use a local cache of data for "tier-1" performance and leave the rest of it in the cloud. There, it's accessible from anywhere, facilitating disaster recovery and global operations as well as production applications.

The key is to make the decision "live in the cloud" - that is, to make a cloud storage service (or multiple services or even a private service) the primary home of data. Once this strategic shift has been made, it changes the data center (and all of IT infrastructure) in a fundamental way. Once data is not tied to physical location, information processing is mobile and flexible. Then the real future of cloud computing appears.

VMware Array Integration is Right Here, Right Now

2012-01-26T22:48:00Z

Virtualization of servers and desktops places a massive strain on traditional storage environments. This is mostly due to the “missing link” of communication between storage arrays and hypervisors. The array needs information about the virtual environment to function properly, and hypervisor could benefit from array features. This is where integration technologies like VAAI come into play, and why these technologies are increasingly important to customers.

Virtual servers and desktops place a unique strain on storage devices. They randomize I/O and increase the amount of traffic on each interface. The hypervisor also interferes with the use of valuable storage features like snapshots, replication, and thin provisioning.

This is the reason that VMware introduced their vStorage API for Array Integration (VAAI) with vSphere 4.1 and the API for Storage Awareness (VASA) in vSphere 5. In both cases, the integration flows in both directions: From the hypervisor to the storage and from the storage to the hypervisor.

The 3 functions of VAAI have become integral parts of modern virtual infrastructure. Many storage arrays have excellent thin provisioning capabilities, but needed better communication about which blocks were no longer required. VAAI exposes this information to supported storage arrays thanks to the Block Zeroing primitive. Arrays also do an excellent job moving data

around, so the VAAI Full Copy primitive allows the hypervisor to offload cloning and mirroring of data.

Since hypervisors are built to share storage resources between clustered servers, a better mechanism was needed to lock and unlock locked storage. This led to the creation of the VAAI primitive known as Hardware Assisted Locking or Atomic Test and Set, which drastically accelerates I/O operations when resources are shared. This type of hypervisor-driven storage functionality is likely to continue to grow in importance in the future, with storage arrays adapting to the demands of virtual data centers.

End-users have been very enthusiastic about the impact of this sort of array integration with the hypervisor. It allows them to take better at vantage of the advanced features of their storage arrays, and results in dramatic performance improvements. Full Copy, for example, can improve the performance of cloning operations by order of magnitude or more, and Hardware Assisted Locking dramatically improves performance of active virtual machine clusters.

If you would like to learn more about these VMware integration features, I suggest you join me on February 15 for a webinar on the subject. Click here to Register. I will be discussing the topic of storage integration for virtual machines with Lucas Nguyen of VMware and Chris Saul of IBM. I hope you can join us. You may also be interested in my recent blog post outlining all of the VAAI primitives in detail.

What IT Pros Need to Know About Cloud Storage, Part 2

2011-12-19T15:55:00Z

One of the biggest challenges in enterprise storage management is under utilization of assets. Even the most advanced storage architecture can be undermined by a capacity planning or purchasing mishap. Certain technologies, like thin provisioning, object storage, and cloud storage can help, but nothing takes the place of good old-fashioned storage management.

Capacity utilization of enterprise storage systems has remained at low levels for decades. IT departments often over by storage capacity when money is available as a hedge against future unfunded demands, and some storage architectures just can't handle high levels of utilization. Then there are the challenges of communication and forecasting of future storage needs. It's no wonder, really, that typical storage environments only have data on about a third of their usable SAN or NAS capacity.

Historic approaches to improving capacity utilization just haven't helped. First came a nonsensical push toward tiered storage, though no one could quite articulate how this improves capacity utilization. Distressingly, many organizations resort to blaming storage administrators for poor utilization, even though they have very little say in how their systems are used.

One cause of low capacity utilization is the block access method itself. Protocols like SCSI and ATA “expect” access to an entire virtual disk drive. Servers connected in this way manage their own pool of capacity outside the traditional sphere of influence of storage and data professionals. Thin provisioning ought to help, but many storage administrators, concerned about running out of capacity, still won't run their systems at high levels of utilization. Even where it is used, thin provisioning is little more than a temporary measure to avoid new storage purchases until usage catches up to the comfort level of IT pros.

But what if a different protocol was used to communicate with storage systems, one that “understood” when files were created, updated, and deleted? Such protocols exist today, in the form of object and cloud storage systems. Although it may require application changes or a gateway to switch storage protocols, organizations are increasingly turning to these alternatives.

In a way, public cloud storage is the ultimate form of thin provisioning. Users can pay only for what they use, with file or object level granularity, and deleted objects can be immediately “given back” to the public pool. Private clouds and object storage systems are conceptually similar, but they still require capacity planning and must be purchased in large chunks. But bigger organizations can get the same kind of on-demand provisioning as public cloud provided they are able to attract a broad base of users.

Utilization has always been a tough nut to crack, so perhaps it's time for IT departments to follow the lead of cloud service providers and begin to transition to object and cloud storage. Applications like archiving and document management can immediately use object and cloud storage systems at high levels of utilization. As these and other applications make the transition, perhaps storage utilization will finally rise from its decade-long slumber.

Read the first article in this series, What IT Pros Need to Know About Cloud Storage, Part 1

Download Storage Community whitepaper, "The Holy Grail of Storage Efficiency."

What Do IT Pros Need To Know About Cloud Storage

2011-12-05T22:38:00Z

IT professionals are decidedly ambivalent about cloud computing generally, and storage pros are downright skeptical about cloud storage. But they needn't be afraid: Cloud storage is not as threatening as it seems, and may even allow storage administrators a little leeway for career advancement!

Let's start with a few definitions: It easiest to understand what cloud storage is if you begin by understanding what is not worthy of the name. Conventional storage systems that use access protocols like SCSI, NFS, and SMB are not “cloud storage” any more than my basement is a video arcade thanks to the Wii and Xbox residing there.

True cloud storage eschews these traditional “stateful” protocols, which include inherent assumptions about location, security, and access control, in favor of HTTP. This allows an additional layer of abstraction, opening the door to new architectures. Most cloud storage systems scale by adding additional nodes rather than scaling by expanding the storage behind an existing controller.

It has been technically challenging to enable conventional storage systems to scale out to hundreds or thousands of nodes in multiple locations, but cloud storage technology can do this without even breaking a sweat.

Most definitions of cloud storage also suggest on demand scaling and as you go pricing. This leads many to assume that all cloud storage is like Amazon S3: Public cloud. But this is not the case. There is no reason that IT cannot build a private cloud storage infrastructure to be shared by many departments. Similarly, there are many options to create a hybrid public/private cloud storage environment.

Although HTTP-based protocols are essential to true cloud storage infrastructure, there's no reason that applications must use these protocols. Indeed, a wide variety of gateway products exist, allowing conventional SCSI or NAS protocols to be used by clients. Many of these include essential business friendly elements like encryption, access control, and data replication as well.

Since cloud storage does not necessarily eliminate conventional applications and protocols, perhaps it also does not eliminate the need for in-house IT staff. Adoption of cloud services does indeed sometimes include handing off infrastructure management to outsiders. They keep everything running, responding to alerts and handling hardware and software maintenance. The cloud service providers do not generally interact with applications or internal business functions.

Enterprise IT desperately needs storage management of a higher level. This is the reason that IT pros should not be afraid of the cloud: It eliminates many of their headaches and allows them to focus on higher-level tasks. Rather than worrying about array firmware or RAID levels, storage managers can truly manage capacity and data usage. They can focus on what is being done rather than how.

SSD is Not the Best Way To Use Flash Memory in Storage

2011-11-21T19:25:00Z

The terms, “SSD” and “flash memory” might seem synonymous to even technical audiences, but they couldn't be more different. Flash memory is the dominant underlying chip technology for solid-state storage. But solid-state disk drives are just one packaging option for flash, and not a very good choice at that. Future storage devices will use flash directly, rather than packaging it to masquerade as a conventional hard disk drive.

Solid-state storage is as old as computing, with DRAM-based storage devices appearing continually since the 1980's. But DRAM was too expensive for general purpose storage, so these high-performance devices were rare and specialized.

Non--volatile flash memory, invented in the 1980's, slowly grew in practicality in the last two decades. Today, single- and multi-cell NAND flash memory has become affordable and practical.

There are many challenges in applying flash memory to conventional IT systems. Existing operating systems and applications expect storage capacity to appear in a conventional block or file format. But NAND flash is not a conventional disk at all, and has completely different region right characteristics. Flash memory excels at random I/O, precisely where rotating disk drives fall flat, so it should make an ideal choice for future storage devices.

The most straightforward approach to applying NAND to conventional IT systems was to place it behind a controller that would mask the complexity of flash, making it appear to be nothing more than a very fast disk drive. This is the definition of a solid-state drive or SSD. It is a compromise that allows unmodified systems to benefit from high-speed flash storage without any significant architectural changes.

But SSD is far from ideal. Each individual drive contains a complicated controller which interprets ATA or SCSI commands and routes data across multiple paths to the flash chips that do the actual storage. A good friend and storage engineer once remarked that “SSD controllers are the world's smallest storage arrays”, and this is pretty close to the truth. These controllers face a daunting challenge of balancing reliability and performance without challenging conventional access methods.

Device makers love SSDs since they can easily swap out a hard drive without radically altering their systems. SSD is a shortcut to the future but does not fully take advantage of the performance or unique characteristics of flash memory.

The compromises inherent in SSD become obvious when one compares these devices to true flash memory storage options. PCIe-based flash memory cards outperform SSDs by a wide margin, since they do not have the bottleneck of a controller and SAS or SATA bus. Specialty storage arrays, designed to take advantage of PCIe-connected flash cards, also outperform SSD-based arrays.

The question of “SSD versus PCIe” is simple: Do you want some performance benefit in a conventional system or a radical upgrade? Enterprise storage is a conservative discipline and has tended to side with SSD in the last few years. But the benefits of “non-SSD” PCIe flash memory are enormous and will become increasingly obvious in the years to come. In a decade, SSD will seem a quaint throwback while flash memory will roar ahead.

Will 16 Gb Fibre Channel Derail FCoE?

2011-10-17T23:04:00Z

FCoE was a hot topic at the Interop conference, but the crowd’s reactions were of skepticism rather than enthusiasm. In both of my sessions, many of the questions focused on the “why” of switching to Ethernet for FC storage rather than the “when”.And the advent of 16 Gb Fibre Channel, much in the news with all major vendors rolling out products this quarter, begs the question: Why use a 10 Gb Ethernet standard that remains in flux when 16 Gb FC is shaping up nicely?

It’s important to recognize that neither 16 Gb FC nor 10 Gb FCoE is really ready for prime time at this point. FCoE is functional as an edge-only protocol, and is gaining traction in specialized use cases like blade servers. But end-to-end FCoE requires integrated Fibre Channel Forwarding and Ethernet fabric technology that remains decidedly experimental, and interoperability is a serious question. The calendar will read “2013” before customers can go out and buy Ethernet switches that are fully capable of plug and play, multi-vendor operation.

16 Gb Fibre Channel is decidedly bleeding edge as well, however. A few HBAs and switches have been announced, but most are only shipping in small volume if at all. And although interoperability among vendors and with previous 8 Gb equipment looks good, it’s not a done deal yet.

But the price premium for 16 Gb FC isn’t that high, and I’m hearing lots of talk about design wins from major OEMs. Analysts expect volume shipment of 16Gb equipment in 2012, though it won't become the majority flavor of FC for another 2 to 3 years after that. A major benefit for both native FC and FCoE is the ease with which they can be added to an existing FC SAN. Both will plug right in, enabling future compatibility right off the bat and faster performance in the future. And a new generation of HBAs are appearing which will work with either standard, allowing customers to buy client-sideequipment now without committing to one or the other in the coming years.

Customers are now beginning their evaluation of next-generation SAN equipment, and will likely allocate major budget spending to 16 Gb FC or 10 Gb FCoE in 2013 and 2014. Which will they choose if both are ready for prime time at that point.Storage is a conservative space since availability and reliability are so crucial to enterprise systems. They will not buy cutting-edge products without a good feeling about supportability.

Vendors are likely to make the difference here. Cisco will likely continue pushing integrated FCoE solutions with EMC and NetApp, but what will HP, Dell, and IBM do? Emulex, QLogic, and Brocade are eager to work with these system heavyweights and all offer solid 16 Gb FC roadmaps. History suggests that customers will side with their systems partners, and that native 16 Gb FC will continue to be attractive in the coming years. They need compelling motivation to switch to Ethernet.

Has the Time Finally Come for Data Reduction?

2011-09-06T16:43:00Z

In an age when compression and optimization of data pervades the lives of consumers, it seems odd that enterprise IT has been slow to adopt the same technology. Compression enables portable music players, digital mobile phones, and HDTV, yet we still store raw “fat” bits on our storage arrays. It’s time for enterprise buyers to catch up with the consumer electronics world!

Imagine if your iPod only held 10 CDs worth of music. Although still somewhat useful, such a device would not be a compelling purchase. But add in MP3 compression in the same device holds 100 CDs and becomes a must-have gadget. Lossy and lossless data optimization technologies make it practical to watch movies on a plane and bring high-definition video and homes. Without compression, media would be stuck in the 1970's!

Enterprise IT is just opening its eyes to the possibilities of compression and data optimization, even though these technologies have been around for a long time. Computer users began employing compression and archiving almost immediately after personal computers were released. I myself employed Stacker to double the capacity of the 20 MB hard drive in my PC, and gzip is a staple on my UNIX machines.

But enterprise storage, on the whole, does not use any sort of compression technology. Enterprise IT relies on full, “fat” bit streams from the SAN to the disk drive. We are willing to spend millions of dollars literally storing nothing, and enterprise storage vendors are perfectly happy to cash those checks!

Initially, it was felt that compression and data reduction would overly impact performance on enterprise storage systems. Indeed, whole scale compression is somewhat CPU intensive. But today's storage systems benefit from the same vast improvements in processing power as desktops. Moore's law has brought compression within reach, if only customers would adopt it.

Thin provisioning has become much more common in enterprise storage systems in the last few years. This can be seen as a sort of first step toward wider acceptance of data optimization technologies, since thinly provisioned storage breaks the strict 1:1 relationship between allocated and usable storage capacity.

Many buyers remain leery of thin provisioning, however, fearing that users will overflow actual capacity and bring systems crashing down. Software vendors like VMware have done much to mitigate these fears, incorporating reporting and alerting features. It is likely that future operating systems will become smarter about alerting, and will follow VMware's lead in incorporating smart “stun” features to mitigate the impact when capacity is truly exhausted.

De-duplication has gained some acceptance, as well, especially for nonproduction data. It would simply not be practical to store backups and archives on disk without de-duplication technology, and most enterprise storage vendors have a solution for these needs. The next step will be incorporating de-duplication technology in primary storage devices, a move that some vendors have already made. Again, however, IT administrators are moving cautiously when it comes to primary de-dupe.

De-duplication of data can be thought of as a special case of data compression, and many are working on improving the efficiency of data storage beyond today's products. Indeed, it is likely that future data optimization technologies will be effective enough to bring the cost of flash memory within the reach of mainstream IT shops. Smaller vendors are already promising this, and we will likely see this combination of data optimization and flash spread throughout the industry in the coming years.

The core issue is not in the technology of data optimization but in the comfort level of enterprise IT buyers. Some audiophiles don't like MP3s, but even they are willing to adopt lossless compression schemes like FLAC, the musical equivalent of thin provisioning, since it enables greater flexibility and portability. It is likely that the benefits of data reduction and optimization technologies will similarly tempt enterprise storage buyers in the future. Then enterprise storage will finally catch up to home entertainment!

Why is Array Integration with VMware So Critical?

2011-08-23T01:00:00Z

Do you enjoy playing games like “telephone” and “charades”? It's fun to guess what other people are thinking when sitting around the campfire or kitchen table. But guessing the motivation and intent of others has no place in enterprise IT. This is why “integration” will become the key feature of IT products for the next decade: Virtualization and cloud are adding layer upon layer to IT infrastructure, and the time has come for communication mechanisms that allow clear messages to cut through the haze. This is the intent of VMware's vStorage API for Array Integration (VAAI), and why I consider it to be one of the most important developments in enterprise storage in the last decade.

VAAI was introduced in vSphere 4.1 and quickly became one of the most commented about features in VMware blogs and user groups. This was not because of some cutting-edge feature in VAAI. In fact, the three basic functions (known as “primitives” in VMware parlance) are technical “nuts and bolts” things. But techies immediately recognized the value of integrating the VMware hypervisor with storage arrays, since storage performance is so critical to server virtualization environments.

It's hard to say which of the three VAAI primitives are most important, but the block zeroing command is perhaps most implemented. This is a communication mechanism that allows VMFS to notify a storage array when space in a VMDK file is no longer needed. A thin provisioning capable array can then reclaim that capacity and use it for some other purpose. VAAI blog zeroing uses either custom interfaces or the standard T10 command that many arrays support.

VAAI full copy can be a real timesaver, allowing the hypervisor to command the storage array to make a mirror of a SCSI LUN rather than reading and writing the entire contents over the storage network. Only custom array interfaces are supported in vSphere 4, but version 5 supports a standard T10 command as well.

The final VAAI primitive is harder for many people to comprehend, since it does not actively cause the array to perform a function. Hardware assisted locking, also known as “atomic test and set” allows multiple virtual machines to share a single SCSI LUN without blocking each other's I/O. This can really smooth performance when clustered virtual machines are in use. The effects are so noticeable that VMware decided to implement similar code throughout vSphere 5.

Expanding VAAI Support

vSphere 5 enhances VAAI in two important ways: additional primitives are added for block storage, and NFS devices are supported for the first time.

On the block storage side (Fibre Channel and iSCSI devices), VAAI now has the ability to reclaim space in VMFS after a storage vMotion or VMDK deletion, and thin provisioning using the standard SCSI UNMAP command is added. VMware also “stuns” virtual machines if a thin provisioning storage array runs out of space.

But the big news for VAAI in vSphere 5 is the addition of NFS support. This reflects the growing importance of NFS as a storage protocol to support server virtualization, with some analysts suggesting it has been implemented even more widely than iSCSI.

VAAI in NFS includes a totally different set of primitives, reflecting the strengths and weaknesses of the protocol. NFS was always excellent at handling thin provisioning, but administrators sometimes would rather reserve capacity. This is the intent of the “Reserve Space” primitive, which instructs a thin array to be “thick.” VAAI for NFS also includes an extended statistics API, allowing vSphere to query arrays about capacity and “thin-ness.”

Two more VAAI-NFS primitives exist, and both are focused on data protection. Full File Clone functions much like Full Copy on block storage arrays, allowing the hypervisor to instruct the array to clone a volume. Finally, Native Snapshot Support will enable VMware View environments to leverage NFS snapshots, though this primitive is not yet implemented elsewhere.

VAAI for NFS in vSphere 5 differs in another major way from the block support. Rather than bundling plugins, VMware leaves it to NFS device makers to distribute their own enablers. It remains to be seen what this means for time-to-market and supportability of VAAI-NFS.

Better Communication = Better Performance

The upshot of all this integration is better storage performance, and this will appear in many forms. Streamlining thin provisioning operations reduces I/O and enables better capacity utilization. Copy offloading functions similarly allow arrays to do what they do best – move and protect data. And all of the VAAI functions are transparent once enabled: They just work. This is perhaps the best feature of all!