Ask an administrator why their backup repository is smaller than the data it protects and most will say compression. It is the intuitive answer, and for a single archive it is the right one. For a backup system holding months of history across dozens of machines, it is close to wrong. The bulk of the saving comes from deduplication, and compression works on what is left over.
The difference comes down to the size of the window each technique can see. A compressor looks for repeated byte sequences inside a search window that ranges from a few dozen kilobytes for a fast codec to at most a gigabyte or so for an aggressive one with a very large dictionary. Anything outside that window is invisible to it. Deduplication has no such limit. Its window is the entire repository, so a block written today can be matched against an identical block stored months ago, from a different machine, in a different backup job. When you take a full backup every night and the underlying data changes by two percent, that distinction is the whole story. Compression will shrink each nightly copy by half. Deduplication will notice that ninety eight percent of tonight’s copy already exists on disk and store almost nothing.
Getting that matching to work reliably is harder than it sounds, and the reason is a problem called boundary shift. The naive approach is to cut the data stream into fixed blocks, hash each one and look the hash up. This works until someone inserts a single byte near the beginning of a file. Every block boundary after that point moves, every block gets different content, every hash changes, and a file that is essentially unchanged deduplicates against nothing. The fix is content defined chunking, where the cut points are chosen by the data itself. A rolling hash slides over the stream and a boundary is declared whenever the hash matches a particular pattern. Insert a byte and only the chunk containing it changes, because the surrounding cut points are anchored to content rather than to offsets.
Chunk size is then the main tuning decision, and it is a genuine trade. Smaller chunks find more duplicates, because matches can be finer grained, but each chunk needs an index entry, and the index has to be consulted for every chunk written. Push the average size too low and the metadata starts to rival the data it describes, while the index stops fitting in memory and every write turns into a random read. Push it too high and near identical files stop matching. Most systems settle on averages somewhere between tens of kilobytes and a megabyte, with the exact choice depending on whether the workload is many small files or a handful of enormous virtual machine images.
The order of operations matters more than almost anything else, and it is the mistake that ruins otherwise sensible designs. Deduplication must happen before compression, and compression before encryption. The reason is that both compression and encryption destroy the surface similarity that deduplication relies on. Two nearly identical inputs produce compressed outputs that diverge completely after the first difference, and encrypted outputs that share nothing at all by design. Compress first and the deduplication ratio collapses to near one. Chunk first, identify the unique chunks, then compress and encrypt each unique chunk individually, and both techniques do their work without interfering.
Once the pipeline is right, the results split cleanly by workload. Long retention of frequently repeated data is where deduplication is transformative. Nightly images of a fleet of similar servers, virtual machine snapshots, workstation backups where half the disk is the same operating system on every machine – these routinely reach ratios that compression could never approach, because the redundancy being removed is between backups rather than inside them. Conversely, deduplication does essentially nothing for a single one off archive, and nothing for content that is already unique and already compressed. A directory of photographs and video files will deduplicate at close to one to one, because every file is different and every file is entropy dense.
That is exactly where compression keeps earning its place. After deduplication has removed the repeated chunks, whatever remains is unique data, and unique data still compresses. Database dumps, logs, source code and configuration files typically shrink by a factor of two to four on the residue alone. The two techniques stack rather than compete, and a system that does both well beats one that does either well.
None of this comes free. A deduplicating repository needs its index resident in memory to stay fast, which turns memory into the limiting resource on the backup server. Restores become random reads, because a single file is reassembled from chunks scattered across the store, and a repository that has been running for a long time fragments in ways that make restore throughput drop well below backup throughput. Deleting old backups is no longer a matter of removing files, since a chunk may be referenced by many snapshots, so the system needs reference counting and a garbage collection pass. And the risk profile changes: one corrupted chunk no longer damages one backup, it damages every backup that references it, which is why checksums, redundancy and regular verification stop being optional.
The practical rule is simple enough. If you are storing one copy of something once, compression is all you have. If you are storing the same thing repeatedly over time, deduplicate first, compress the remainder, and expect the first step to do most of the work.