Per-tenant encryption at rest for HBase backups

Problem

Tenant-level encryption lets customers encrypt their data at rest with their own keys, much like S3's SSE-C, where the storage service does the encryption and the customer supplies the key. HBase already applies it to its store files, but backups are written in a different format. They were encrypted at the disk level, so they were never stored in the clear, but the customer's key didn't protect them.

The goal was to encrypt backups with the same per-tenant keys HBase already manages, with no second keystore and no new operator steps. Restores had to stay parallel and keep working after a key rotates, tenants without a key keep the current format, and old backups still restore.

Context

A backup is a set of large chunk files, each holding (row key, row) records. Next to each chunk sits a manifest with one entry per tenant, which restore uses to find a tenant without opening chunks. The design depends on one property: the writer sees each tenant's rows together, one tenant after another. Today a chunk usually holds many tenants, and a tenant can span several chunks.

HBase already has a key hierarchy, and the design reuses it. A system key per cluster protects the tenant keys at rest. Each tenant has a data key (DEK), issued by an external key management service, which can be active, inactive (still able to decrypt) or disabled. The key that actually encrypts data, the content key (CEK), is derived from the DEK on demand and never stored. Protecting data keys with a higher-level key this way is called envelope encryption: data keys are only ever stored wrapped.

The cipher is AES-GCM. Besides encrypting, it produces an authentication tag, a short checksum that only the key holder can compute and that decryption checks. Each encryption also takes an IV (initialisation vector), a value that must never repeat under the same key.

Approach

Keeping backup keys apart from store-file keys

The encryption key is derived from the tenant's data key with HKDF, a standard key-derivation function (RFC 5869). HKDF takes a context label, and keys derived with different labels are independent. HBase used the same label for everything, so in one of its key modes a backup would have encrypted under exactly the same key as the tenant's store files.

That matters because GCM breaks badly if the same key and IV are ever used twice: an attacker can recover the XOR of the two plaintexts and forge tags (NIST SP 800-38D, Appendix A). Sharing a key between two systems that pick IVs independently makes that collision possible. The fix is a separate label for each use, one for store files and one for backups.

A separate chunk for each encrypted tenant

The main choice was how much data each key covers. The options differ most in what happens when one tenant's key rotates or is destroyed:

We started by giving every tenant that has a key its own chunk files. Every chunk is then encrypted under exactly one key, which keeps the format and the restore path simple. Sharing chunks again, with one key per tenant's block, is the later step if the number of small files becomes a problem.

Writing and restoring an encrypted backup

Backup. Encryption happens in the output writer, after rows are sorted, because ciphertext looks random and encrypting earlier would destroy the row order that tenant contiguity depends on.

1. When a new tenant's rows begin, the writer closes the current chunk and starts a new one for that tenant. The manifest records which key each chunk uses. Tenants without a key, and rows that belong to no tenant, such as global metadata rows, go into unencrypted chunks as before.
2. The whole chunk is encrypted with AES-GCM under a single CEK, derived from the tenant's DEK, with one IV for the chunk. Records are compressed before they are encrypted.
3. One IV per chunk adds no per-record overhead.

Restore.

1. A restore task reads the chunk's key from the manifest and decrypts the chunk from the start. A chunk is one encrypted unit, so restores parallelise across chunks instead of within one.
2. The GCM tag covers the whole chunk. If it doesn't match, the task fails instead of skipping data.
3. The same check covers the wrong key, whether from a bug, a stale manifest or tampering: decryption with the wrong key fails the tag check, except with negligible probability, so the manifest needs no separate integrity check of its own.

Later, with shared chunks. Row keys are encrypted as well, so once tenants share a chunk, reading a row doesn't reveal whose it is. Each tenant's block would get its own CEK and IV, the writer would record each block's start offset in the manifest, and the reader would switch keys at those offsets. A wrong offset is caught the same way, by the tag check.

Handling key lifecycle events

Backup never triggers a key lifecycle event. It only has to handle each one correctly:

Destroying a key is also how one tenant's data leaves retained backups without rewriting anything. Old keys have to be kept for as long as any backup that needs them is retained, and if the key service is unreachable the backup fails instead of writing unencrypted data.

Result

The design extends existing tenant keys to backups without adding another keystore, keeps restores parallel across chunks, supports key rotation without rewriting old backups, and limits a key's destruction to one tenant's backups.

Building encryption at rest means reasoning at several layers at once: the storage format, which decides where keys can switch and how restores split the work; the choice of cipher and how its keys and IVs are derived; and how key rotation, disablement and destruction affect backups that are already written.