Data processing engines · 8 min read
One backup job on MapReduce, Spark and Flink
Our HBase backup ran as a daily MapReduce batch job, and we wanted it much closer to real time. As an intern in 2018 I ported the job to Spark. In 2026 I co-mentored a Flink version that runs continuously, with a target recovery point of 5 to 15 minutes. This writeup compares how the three engines handle output, retries and committed data.
On this page
Problem
Our HBase backup is a daily batch job: MapReduce reads a snapshot of each table and writes every tenant's data to S3. We wanted to modernise it and bring backups much closer to real time.
A batch job sets the recovery point. Anything written since the last committed backup is not backed up yet, so it is lost if the data is lost. Running the job more often doesn't fix that, for two reasons. Every run has a cold start: a new MapReduce job has to be scheduled and its containers started before any data moves. And every run contends for resources with the other jobs on the cluster. Getting close to real time needs a job that runs continuously.
Context
Backups are stored per tenant: each tenant's rows go to their own files, split into chunks, and each chunk has a manifest with row counts and a content hash. A finished backup also contains a success marker file and a metadata properties file. Every port has to write exactly this layout so that restore keeps working.
The MapReduce job reads a table snapshot straight from HDFS, one mapper per region, so it puts no load on the HBase servers. For salted tables, where Phoenix prefixes every row key with a hash byte, a reducer is needed to remove the salt and group the data by tenant.
Approach
Spark, 2018
I rebuilt the job as a Spark DAG with a stage for the map side and a stage for the reduce side. The main goal was to keep processing in memory, where MapReduce writes map output through a disk-backed sort before the reducers read it.
Reading mapped cleanly. HBase's Spark connector exposes a table scan as an RDD with one partition per region, the same split MapReduce used. Writing was the hard part. The existing writer, which gives each tenant its own files, depends on bookkeeping that every MapReduce task has and a Spark task doesn't: a task-attempt ID and an output committer. I recreated that bookkeeping inside each Spark task, so the existing writer and its commit step ran unchanged.
Flink, 2026
Spark Structured Streaming could run the same scan every few minutes as a micro-batch, but each batch is a new job that has to be scheduled, work out region boundaries again and open new scanners. At 15 minutes that overhead is small. At 5 minutes it is a real share of each cycle. Flink deploys its operators once and keeps its HBase connections for the life of the job, and for the shorter target that was the deciding difference.
The pipeline is a straight line: scan HBase, parse and filter tenants, write. There is no shuffle, and the files match what MapReduce produces.
The job cuts time into fixed windows over cell timestamps, five minutes by default, and backs up one window per cycle once that window is entirely in the past. Flink's checkpoints drive the cycle. A checkpoint is a consistent snapshot of the whole job, and after a failure the job restarts from the last one.
Each window's output goes to a hidden folder, which is renamed to its final name only after the checkpoint covering it has completed everywhere. This is Flink's two-phase commit, and the rename is the commit. If a worker fails before the rename, the hidden folder is discarded and the window is scanned again. If it fails after the files are written but before the rename, the rename runs on recovery, because the pending commit was saved in the checkpoint. The first version had a bug here: parallel writers race to rename the same folder, and a loser's check for an already-committed window also matched the still-hidden folder, so it could report a commit that hadn't happened.
Streaming also multiplies small files. With five-minute windows one table produces at least 2,304 files in 288 folders a day, and the HDFS NameNode keeps every file in memory. A combiner periodically merges a run of windows into one ordinary backup folder, validates it, and only then deletes the small files.
On a cold start the job begins where the last successful backup ended, whether MapReduce or Flink wrote it, so switching over leaves neither a gap nor an overlap.
Event time is not processing time
Every HBase cell has a timestamp: the time the write happened. That is event time. The job backs up a five-minute window once that window has passed on the job's own clock. That is processing time.
The two don't always match. A write can be stamped 10:03 but only become visible to the backup scan minutes later, for example when a server fails and its pending writes are replayed after recovery. If the 10:00 to 10:05 window has already been backed up by then, that cell is never backed up.
Flink handles this with event-time windows and watermarks. A watermark is a marker that says event time has reached a given point, so no older data is expected. A window closes when the watermark passes its end, not when the clock does, and anything that arrives after that counts as late. Processing-time windows have no idea of late data: a delayed event lands in whichever window is open when it arrives.
Our proof of concept had no real watermark. It used each window's end time as the watermark, so a window closed the moment the clock passed its end, with no room for late cells. A proper watermark would trail the clock by more than the worst expected delay before a window is scanned. That makes the recovery point later by the same amount, and a cell delayed beyond it is still missed, so the delay has to cover the worst case.
Recovery point and catch-up
Two settings decide how fresh the backup is: the window width and the checkpoint interval. We set the defaults to five-minute windows and one-minute checkpoints, and allowed only one checkpoint at a time, so each checkpoint covers exactly one window scan.
In the worst case a write lands at the very start of a window. It waits for the window to close, then for the next checkpoint, then one more cycle while the window is scanned and committed. With these defaults that is about seven minutes plus the time a checkpoint takes, inside the 5 to 15 minute target. Most of it is the window itself, so the window width is the setting to change for a fresher backup.
After a cold start the job also has to catch up from wherever the last batch backup ended. It backs up one window per cycle, so it gains on real time only while a cycle is shorter than a window. A cycle lasts as long as the longer of the checkpoint interval and one window's scan. Keeping checkpoints at one minute against five-minute windows means the job catches up in about a quarter of the time it started behind, as long as a window's scan also finishes within a minute. If a cycle ever takes longer than a window, the job falls further behind for good.
The three engines side by side
| MapReduce | Spark | Flink | |
|---|---|---|---|
| Lifetime | One run per schedule; a new JVM per task | Executors live for the application; a new job per batch or trigger | Operators deployed once, as threads in long-lived TaskManagers |
| Regions | One mapper per region, fixed at job start | One partition per region, recomputed every batch | Key ranges assigned to subtasks once, at startup |
| Data exchange | Mapper output sorted and spilled to disk, pulled by reducers | Pull-based shuffle, mostly in memory, spilling when full | Pipelined, with credit-based flow control between operators |
| Failure | Rerun the failed task | Recompute lost partitions from lineage | Roll every operator back to the last checkpoint |
| Commit | Output committer; success marker at job end | Hadoop committers at the end of each batch | Two-phase commit; pending folder renamed when the checkpoint completes |
| Slow sink | Tasks run longer | The batch runs longer and the next trigger starts late | Buffers fill and the source pauses |
What this job uses from Flink is a long-lived source, checkpoints to cut the stream, and the two-phase-commit sink. It doesn't yet use watermarks, and it has no large keyed state, two of the usual reasons to choose Flink: its whole state is a cursor, a cycle counter and the open transaction.
Result
The Spark port worked end to end: we ran a full backup and restore test with it and benchmarked its memory use. The Spark and Flink versions are both proofs of concept, and we want to adopt them eventually. To check correctness, the same time range was backed up with Flink and with MapReduce, and the two backups held identical data.
Four things stand between the Flink version and adoption. Its scans put read load on the HBase servers, which Phoenix CDC or tailing the write-ahead log into Kafka would remove. Late cells need a watermark, as described above. Its rename-based commit isn't safe on S3; we evaluated Flink's own S3 connector but didn't build an S3 version. And always-on jobs need a Kubernetes controller in place of cron.