Operate

Disaster recovery

The real restore, not the test. How to turn one of Restorable's encrypted backup blobs into a running database when you actually need it.

A restore test is Restorable's job. A production restore is your job; Restorable gives you the encrypted backup, the key, and the procedure. The receipts you have accumulated are what let you pick the right backup to restore.

Restorable deliberately does not write to your databases. There is no restorable restore --to <target>. The agent decrypts on the recovery host and stops at "dump file on local disk." You apply it with your engine's tool, with your flags, on your timing.

Decide which backup

In the dashboard, open Receipts. Find the most recent passing receipt for the source you are restoring. The receipt records:

  • The backup_id the test ran against. This is the ciphertext you want.
  • The backup.created_at timestamp. When the backup was taken.
  • The summary-row JSON the check returned at that backup's restore. If that row looks like the data you need, you have the right backup.

The most recent passing receipt is usually the right choice. If you are restoring because of data corruption that predates some recent backups, walk back through receipts until you find the last one whose summary-row looks healthy for your use case.

Pull the dump

Two paths, depending on what you have on the recovery host.

From the agent host (or any host with the keys)

If your agent's install root is intact (or you've copied keys/ onto a fresh host and run restorable init there), the agent decrypts in one command:

restorable download \
  --backup bkp_01hz... \
  --out ./restores/

The agent streams the ciphertext from the orchestrator, verifies the SHA-256 against the registered metadata, decrypts with your age key, and writes <backup_id>.pgdump (Postgres) or <backup_id>.mongoarchive (MongoDB) into the directory you supplied. Plaintext never lands on disk before the verified write. The summary line at the end shows the next-step command you need.

To pull the most recent backup for a source instead of a specific id:

restorable download --latest --source prod-db --out ./restores/

From a bare host (no agent installed)

If the agent host is gone and you're working on a fresh machine, open the backup's detail page in the dashboard and click Mint download URL. The orchestrator issues a 15-minute pre-signed S3 URL scoped to that one object and renders the three-command runbook with copy buttons:

curl -fsSL -o "bkp_01hz....ciphertext" '<short-lived-url>'
age -d -i ./age.key < "bkp_01hz....ciphertext" > "bkp_01hz....pgdump"
pg_restore --dbname "$TARGET" "bkp_01hz....pgdump"

No Restorable binary required on the recovery host. The URL expires in 15 minutes; mint another from the dashboard if you stall.

Decrypt with your age key (manual variant)

If you ran restorable download --ciphertext (or you used the bare-host curl flow above), you have a raw age-encrypted blob. Decrypt:

age -d -i /var/lib/restorable/keys/age.key \
  < bkp_01hz....ciphertext \
  > bkp_01hz....pgdump

The output is the engine-native dump: pg_dump --format=custom for Postgres, mongodump --archive for MongoDB. No additional unwrap step.

If you encrypted the backup under a previous CEK (key rotation), check cipher.recipients_hash in the receipt against the rotation history in Settings → Keys. Point --root (or the -i flag on age) at the key file that was active when the backup was taken.

Restore, for real

Set $TARGET to the connection string for the database you are restoring into. The base command is the same one the agent's download summary and the dashboard's runbook print: identical across every surface.

For Postgres:

pg_restore --dbname "$TARGET" ./restores/bkp_01hz....pgdump

For MongoDB:

mongorestore --uri "$TARGET" --archive=./restores/bkp_01hz....mongoarchive

Restore into a fresh database first, validate, then cut over. Restoring on top of a live source overwrites everything that happened since the backup was taken.

Common flags worth knowing

Layer these on as your context demands. Restorable does not decide for you; your runbook does.

Postgres

  • --single-transaction: either the whole restore succeeds or nothing lands. Also blocks untrusted procedural- language definitions from executing mid-restore.
  • --no-owner --no-privileges: ignore ownership and grant statements. Right when restoring into a fresh database with different roles.
  • --jobs N: parallelize across N worker processes. Tune to your target's I/O ceiling.
  • pg_restore -l file > toc.txt then pg_restore -L toc.txt … to restore selected tables only.

MongoDB

  • --drop: clear target collections before restoring. No-op on a fresh database; destructive on one with data.
  • --nsInclude / --nsExclude: scope the restore to specific namespaces (db.collection patterns).

Verify you restored the right thing

Run the same check the receipt recorded, against the restored database. The receipt's checks[] array records the SQL / Mongo find and the summary row it produced at restore-test time; re-running against the real restore should produce something close (exact if the data has not changed since the backup was taken).

This is the "trust but verify" step: you trust Restorable because of the receipt, and you verify the restore matches by re-running the check manually.

Shred the decrypted copy

The decrypted dump is cleartext customer data. Do not leave it lying around:

shred -u ./restores/bkp_01hz....pgdump ./restores/bkp_01hz....ciphertext

Document the procedure in your runbook.

When the agent host itself is gone

If the agent host is the thing that died, you have two paths.

Bare-host curl flow. Use the dashboard's Mint download URL on the backup detail page and follow the three-command runbook above. Needs only curl, age, and your engine's restore tool on whichever machine you're recovering on.

Reinstall the agent. If you prefer the one-command restorable download path:

  1. Restore the install root on a fresh host per Backing up the install root. The keys at keys/age.key and keys/signing.key must be the ones that were active when the backup was taken.
  2. Run restorable init to write a fresh config.yaml against the orchestrator. Your signing identity is preserved if you reuse the keys.
  3. Run restorable download as above.

Without the off-host install-root backup, the CEK is gone and so is the ability to decrypt. This is the category of loss the install-root backup page is written to prevent.

Time budget

A rough sketch of wall-clock time on a small production DB:

  • Pick receipt in dashboard: 1 minute
  • restorable download (or curl + age): 1–5 minutes, dominated by transfer
  • pg_restore into live DB: minutes to hours, scales with dump size
  • Verify check against restored DB: under a minute

The bottleneck is almost always pg_restore. If you expect to do real restores under RTO pressure, a restore test at the same scale is the best rehearsal.