Moving backups to S3 without the rename penalty

Problem

Our HBase backups were written to HDFS on each cluster. Object storage costs less and suits backup data, and moving to it was a step towards backups that run on any cloud's storage with few changes. On S3, though, the same backup job ran far longer, and before rolling out we had to find where the time went.

The test bed was a large pre-split test table, six NodeManagers (the YARN worker nodes that run tasks), and a backup job of several hundred mappers. The same job ran once against HDFS and once against S3:

The mappers took about the same time on both. Almost all of the extra time was spent after the last mapper had finished.

Context

The backup service had been written against HDFS, and paths were assembled in many places with nothing saying which filesystem they belonged to. Before any benchmarking I changed every backup, merge, restore and deletion job to use fully qualified paths through Hadoop's FileSystem API, with S3A behind it. That refactor was dull, but it is what made the later rollout a configuration change.

The harder difference was semantic. HDFS renames and deletes a directory atomically, as one metadata operation. S3 is an object store with neither: a rename is a copy and a delete of every object, and a directory delete removes objects one at a time, so both are slow and either can stop halfway. Hadoop jobs written for HDFS rely on both, so moving to S3 needed changes to how jobs commit and clean up, not just a new path.

Approach

Adding nodes didn't help much

My first guess was capacity: perhaps S3 needed more parallelism to hide request latency. I reran the backup with more NodeManagers and also tried larger multipart upload sizes.

More nodes helped only a little and stopped helping at 15. Commit time barely moved, because commit was not running on the NodeManagers.

Where the time went

An output committer is the part of a Hadoop job that moves finished task output into its final location, so that failed or retried tasks leave nothing behind. Hadoop 2.10's default committer does this with renames. Each task writes to a temporary directory, and at the end of the job one process renames everything into the final directory, one directory at a time, on a single thread.

On HDFS that costs almost nothing. On S3 every rename is a copy, so job commit turned into a serial copy of the whole backup on one thread. That was the 3h 25m, and it is why extra nodes did little for it.

The fix belonged in the commit protocol. Hadoop 3's S3A committers avoid rename completely: tasks upload their output to the final location as multipart uploads, and job commit only marks those uploads complete. The data is uploaded once, and finishing the job takes one small request per file. I reran the benchmark with one of them:

What we could ship on Hadoop 2

Production was on Hadoop 2.10, and a move to Hadoop 3 was some way off. I wrote down three options:

• Backport the S3A committers into our Hadoop 2 fork.
• Write final paths directly with a no-op committer, which moves all the failure handling into our own code.
• Switch the default committer to its version 2 algorithm.

Version 2 renames each task's output straight into the final directory when the task finishes. The bytes copied are the same, but the copies are spread across all the tasks instead of queued on one thread at the end. In my tests it came within 3–4% of the S3A committers, and it needed only a configuration change. The cost is that a failed job can leave part of its output in the destination. Backup folders without a success marker were already treated as failed and cleaned up, so that partial output already had somewhere to go.

Rolling it out without a copy

The obvious migration was to copy the existing backups to S3 with DistCp and then switch the jobs. I argued against it: it was a large one-off copy with no clean way back, DistCp is itself a MapReduce job with the same commit problem, and retention would expire most of the copied data soon anyway.

Instead, jobs read backups from both the old and the new filesystem and write only to the new one, so restores and merges see one combined history. As retention ages out the HDFS backups, the old location empties without anyone copying it. The storage target was a single flag per cluster, so rolling back meant pointing the writer at HDFS again. We rolled out in stages: a development cluster, then one production cluster, a DR cluster a few weeks later, and the rest after that.

Other jobs that rename and delete

The committer change covered the backup write path, but merges, repairs, retention and cleanup jobs still renamed or deleted directories. On S3 each of these is a series of object operations that can stop halfway, so we made each one safe to retry.

We also considered replacing our retention deletion with S3 lifecycle rules, and kept our own. Our deletion keeps any backup that a point-in-time restore inside the retention window still needs, which an age-based rule can't express.

Result

We shipped committer version 2 on Hadoop 2.10, and on Hadoop 3 the S3 path moved to the magic committer, one of the S3A committers. When we built backups on GCP, we did the same work for Google Cloud Storage, where the connector has no equivalent of the magic committer: we benchmarked the Manifest committer, which runs job commit's copy and delete work on a thread pool, against the default committer. The general takeaway is to let a benchmark make the decision: here it ruled out adding nodes and picked the committer we shipped.