Backing up the install root
The agent's install root holds the keys that make your history verifiable. If the host dies and nothing else has a copy, every past receipt becomes unverifiable and every past backup undecryptable. This is the one category of loss Restorable cannot recover from for you.
Shamir escrow, which would let a quorum of custodians reconstruct the keys without any one party holding them, is on the roadmap. Until it ships, a periodic off-host backup of the install root is the only safety net. Plan for it.
What is in the install root
Default location: /var/lib/restorable/. Owned by the
restorable user at mode 0700.
| Path | Recoverable without a copy? |
|---|---|
keys/signing.key | No. Losing it makes every past receipt unverifiable. |
keys/age.key | No. Losing it makes every past backup undecryptable. |
keys/api_token |
Yes. Rotate the token from the dashboard and re-run
init.
|
blobs/ | Yes. Ciphertext queued for upload. Dropped on next scheduled backup. |
receipts/ | Yes. Local cache; the orchestrator holds the canonical copy. |
events/ | Yes. Local cache of agent-side events. |
Two files matter. keys/signing.key and
keys/age.key. Everything else the orchestrator can
help you rebuild.
The minimum viable backup
A single off-host copy of /var/lib/restorable/keys/.
Encrypted at rest, stored somewhere the agent host cannot reach
on its own.
On the agent host, as root:
umask 077
tar -C /var/lib/restorable -czf /root/restorable-keys-$(date +%Y%m%d).tar.gz keys/
gpg --output /root/restorable-keys-$(date +%Y%m%d).tar.gz.gpg \
--symmetric --cipher-algo AES256 \
/root/restorable-keys-$(date +%Y%m%d).tar.gz
shred -u /root/restorable-keys-$(date +%Y%m%d).tar.gz
Move the .gpg file to somewhere durable. A second
cloud provider. An encrypted USB stick in a safe. An offline
machine. Anywhere that survives the loss of the agent host and
is not reachable from the agent host.
The passphrase you chose for gpg --symmetric is now
as important as the keys themselves. Treat it the way you treat
your password manager's master password.
Frequency
The keys only change when you rotate them. If you do not rotate, one off-host copy at install time is correct forever. If you rotate on a cadence, take a new copy after each rotation.
The rest of the install root (blobs, receipts,
events) changes constantly and is not worth backing
up. The orchestrator holds authoritative copies of everything
that matters.
Restore procedure
On a fresh host, install the agent normally. Do not run
init. Instead, recover the keys from your backup:
# On the old machine, decrypt:
gpg --decrypt restorable-keys-YYYYMMDD.tar.gz.gpg > restorable-keys.tar.gz
# Transfer restorable-keys.tar.gz to the new host via a secure
# channel. Delete the plaintext tarball from the old machine
# afterwards.
# On the new host, as root:
sudo install -d -o restorable -g restorable -m 0700 /var/lib/restorable/keys
sudo tar -xzf restorable-keys.tar.gz -C /var/lib/restorable
sudo chown -R restorable:restorable /var/lib/restorable/keys
sudo chmod 0600 /var/lib/restorable/keys/signing.key
sudo chmod 0600 /var/lib/restorable/keys/age.key
shred -u restorable-keys.tar.gz
Then rotate the agent's API token in the dashboard and run
init with the new auth key. The agent comes back
online with the same signing identity, so the chain of past
receipts stays intact.
The old host's api_token is invalidated by the rotation, so if the old host is still alive somewhere it cannot submit anything to the orchestrator.