Lifecycle

Backing up the install root

The agent's install root holds the keys that make your history verifiable. If the host dies and nothing else has a copy, every past receipt becomes unverifiable and every past backup undecryptable. This is the one category of loss Restorable cannot recover from for you.

Before you read on

Shamir escrow, which would let a quorum of custodians reconstruct the keys without any one party holding them, is on the roadmap. Until it ships, a periodic off-host backup of the install root is the only safety net. Plan for it.

What is in the install root

Default location: /var/lib/restorable/. Owned by the restorable user at mode 0700.

Path Recoverable without a copy?
keys/signing.key No. Losing it makes every past receipt unverifiable.
keys/age.key No. Losing it makes every past backup undecryptable.
keys/api_token Yes. Rotate the token from the dashboard and re-run init.
blobs/ Yes. Ciphertext queued for upload. Dropped on next scheduled backup.
receipts/ Yes. Local cache; the orchestrator holds the canonical copy.
events/ Yes. Local cache of agent-side events.

Two files matter. keys/signing.key and keys/age.key. Everything else the orchestrator can help you rebuild.

The minimum viable backup

A single off-host copy of /var/lib/restorable/keys/. Encrypted at rest, stored somewhere the agent host cannot reach on its own.

On the agent host, as root:

umask 077
tar -C /var/lib/restorable -czf /root/restorable-keys-$(date +%Y%m%d).tar.gz keys/
gpg --output /root/restorable-keys-$(date +%Y%m%d).tar.gz.gpg \
    --symmetric --cipher-algo AES256 \
    /root/restorable-keys-$(date +%Y%m%d).tar.gz
shred -u /root/restorable-keys-$(date +%Y%m%d).tar.gz

Move the .gpg file to somewhere durable. A second cloud provider. An encrypted USB stick in a safe. An offline machine. Anywhere that survives the loss of the agent host and is not reachable from the agent host.

The passphrase you chose for gpg --symmetric is now as important as the keys themselves. Treat it the way you treat your password manager's master password.

Frequency

The keys only change when you rotate them. If you do not rotate, one off-host copy at install time is correct forever. If you rotate on a cadence, take a new copy after each rotation.

The rest of the install root (blobs, receipts, events) changes constantly and is not worth backing up. The orchestrator holds authoritative copies of everything that matters.

Restore procedure

On a fresh host, install the agent normally. Do not run init. Instead, recover the keys from your backup:

# On the old machine, decrypt:
gpg --decrypt restorable-keys-YYYYMMDD.tar.gz.gpg > restorable-keys.tar.gz

# Transfer restorable-keys.tar.gz to the new host via a secure
# channel. Delete the plaintext tarball from the old machine
# afterwards.

# On the new host, as root:
sudo install -d -o restorable -g restorable -m 0700 /var/lib/restorable/keys
sudo tar -xzf restorable-keys.tar.gz -C /var/lib/restorable
sudo chown -R restorable:restorable /var/lib/restorable/keys
sudo chmod 0600 /var/lib/restorable/keys/signing.key
sudo chmod 0600 /var/lib/restorable/keys/age.key
shred -u restorable-keys.tar.gz

Then rotate the agent's API token in the dashboard and run init with the new auth key. The agent comes back online with the same signing identity, so the chain of past receipts stays intact.

The old host's api_token is invalidated by the rotation, so if the old host is still alive somewhere it cannot submit anything to the orchestrator.