How Data Domain de-dupe is upping its nearline capabilities

Storage de-duplication
has the potential to be used in lots of situations—and de-dupe specialist
Data Domain is having to work hard to prioritise provision of new features from
the opportunities it is seeing.

The starting point
is using its NAS-style de-duplication storage appliances which can be installed with minimum
disruption to an organisation’s existing way of working. This means that, for
instance, it carries out an in-line de-dupe transparently within an unchanged
backup procedure. The company says this will typically achieve an immediate 20x
backup disk saving and requires no management.

So my question is:
“Why wouldn’t you?” Yes, you have to pay for the de-dupe appliance but the massive
disk capacity savings achieved means avoiding future disk drive purchases. In
turn this can, for instance, greatly defer the day when your data centre runs
out of capacity (space, energy) so it also fits well with a green IT policy.

Data Domain also
uses this de-dupe process for a virtual tape library (VTL). The huge disk
capacity saving means data can be economically retained on disk—nearline storage—for, perhaps, months before there is a need for it to go into deep tape (or
optical) archive. In the meantime it is much more rapidly recoverable and
accessible. With the data taking, say, 1/20th the capacity on low
cost SATA disk compared with ‘un-deduped’ tape, the economics of disk versus
tape is radically altered in disk’s favour.

In both cases the
data is accessible reasonably fast, so it provides a nearline tier which can be
accessed directly for many applications; for instance Data Domain has
partnerships with a couple of content search engine providers. Storage content searches
are useful as input to discovery as evidence for a compliance court case.

A new Data Domain
feature is Retention Lock; this can set a lock on individual files as they are
archived so that they cannot be changed in any way for a pre-set period. Since
this is open for the IT manager to set or change it is not suited to rigorous SEC-level
compliance, but helps ensure good governance since it will firmly block user
access. The company also uses a partner to provide encryption. Together these
steps show Data Domain making at least tentative moves into accommodating governance,
risk and compliance (GRC) needs. A data destruction verifiable delete facility
is also planned this year.

In fact de-dupe is
equally at home with archiving as with backup, although the nature of archiving
means the space saving of, typically 75–80% or 4x, is much lower than for
backup; but it’s still impressive. Moreover, the process is also helping remove
the demarcation between backup and archive systems which, at least longer term,
should help simplify the management process.

Further ways this
is supported is that sending either a backup or archive copy to a remote
location, even travelling over a WAN, is practical. Now add a frequent snapshot
capability which sends hardly any data as it only needs to store data tags, and
you nearly have continuous data
protection (CDP) and a very low-cost disaster
recovery (DR) solution. You also obviate any need to physically transport newly-created
tapes to a remote secure location—by sending the information over the wire.

All these are
possible only because the specially-designed appliance, which draws heavily on
CPU performance, achieves the necessary throughput to carry out block- and
byte-level de-dupe in-line as the data is received. Any vendor providing only a
software solution cannot achieve this throughput—and building an optimised appliance
is not an overnight job. The alternative, so-called ‘post-processing’ de-dupe
that only works on the already backed-up storage, has very little value in my
book, as it needs to allocate more disk
space and incurs extra management.

So, notwithstanding
the economic downturn and with storage volumes set to continue soaring, Data Domain
looks to be sitting pretty right now.

What of the
future? Clearly, since applications can already access de-duped nearline
storage in real time, there are few technical reasons stopping de-dupe being
applied to tier one (even tier zero) storage and saving yet more space—except
in considering when to accomplish the de-dupe. (No immediate plans for this I’m
told.) What I do know is that Data Domain’s own users are thinking outside the (storage)
box to pass on their ideas—so some highly original future developments are
entirely possible.