Back up, restore, and prove it

This page is for the person who owns the server the record lives on. It has three acts, each a command you run yourself: write a backup, prove the backup restores, and watch the health line that tells you when backups have stopped. A backup nobody has restored is a promise; the drill turns it into a measurement.

What a backup holds

saphan backup writes one directory, <backup dir>/<UTC timestamp>/, while the console keeps serving.

A piece that does not apply is written down in the manifest, never silently left out. The bundled git server's own data is not part of the backup today: the manifest lists the forge piece as not-applicable. Back up the git server's volumes with your host-level backup.

⚠ Limits that are part of every backup — the manifest repeats them as remarks:

• The store is one snapshot. CouchDB is read page by page while it keeps serving; a read during which the database changed is discarded and taken again (three attempts), then the piece fails by name.
• A PostgreSQL store is checksummed but never restored by the drill: the drill reports not-proven: postgres restore not implemented and never prints PASS for it. Prove a PostgreSQL dump by hand with pg_restore into a disposable database.
• Files in the state directory outside the store — the console login, keys — are not in this backup. Keep them in your host-level backup.

1. Choose where backups go

Backups must not live on the disk they protect. Point them at another volume and decide how many to keep:

saphan config set backup.dir /srv/saphan-backups
saphan config set backup.keep 14
saphan config set backup.schedule 23:00

2. Write a backup

saphan backup --workspace <workspace>

The command prints one row per piece and ends with a RESULT: line. RESULT: ok means every piece was written. If a piece failed, the directory is kept with a .failed suffix, RESULT: error names the piece, and that directory is never counted as a backup and never causes an older one to be removed.

3. Prove it restores

saphan restore --drill /srv/saphan-backups --workspace <workspace>

Name one backup directory, or the backup directory itself, in which case the newest good backup is used. The drill:

1. verifies the manifest's signature, then every file's checksum and size, and stops at the first mismatch;
2. restores into a fresh scratch directory — never into your install;
3. recounts the rows of every table, the documents of every CouchDB dump and the refs of every repository against the manifest, and runs SQLite's own integrity check on the restored store;
4. starts a console server on the scratch copy, on a loopback port with no outbound connection, and asks it one read;
5. prints PASS, UNMEASURED or FAIL (with the first mismatch by file and piece);
6. removes the scratch directory, unless you pass --keep.

The signature. When the host holds a machine device key (created by saphan machine admit --mint-device-key), saphan backup signs the manifest, and the drill refuses a manifest that does not verify under that key. The key is never stored in the backup, so someone who can rewrite the backup directory can rewrite the checksum list but cannot sign it. A host with no device key writes an unsigned backup: its drill prints UNMEASURED, never PASS.

⚠ What a PASS does not say: the drill starts no CouchDB, so a CouchDB dump is proved by checksum and document count, not by loading it into a node. Run the drill after every change to what you back up, and at least once a month.

4. Schedule it

saphan service install registers the backup schedule beside the other services: a systemd timer on Linux, a launchd job on macOS. saphan service status lists it and saphan service uninstall removes it. If you change backup.schedule, run saphan service install again.

5. Watch the health line

saphan fleet health prints one line for backups:

backup: ok — newest 2026-10-29T23:00:04Z (3h0m0s old; red past 48h0m0s)

The line turns red when the newest good backup is older than twice the schedule period (48 hours), when no backup exists, or when the newest backup is not usable — a failed run, or a recorded file that is missing or of another size. It reads the backups themselves, not the scheduler, so a schedule that was removed or is silently failing turns it red the same way. saphan fleet health --json carries the same reading under backup.

⚠ The health line checks sizes, not contents, and retention does not check signatures. Only the drill, on a host that holds the device key, proves a backup has not been rewritten.

6. Restoring for real

saphan restore offers the drill and nothing else: restoring into a running install would overwrite what is running, so no command does it. On the machine you are rebuilding:

1. Stop the Saphan services and keep the old state directory aside.
2. Copy store/saphan-studio.db from the backup into <workspace>/.saphan/state/. A PostgreSQL backup is store/saphan-studio.pgdump; load it with pg_restore.
3. For each bridge/repos/<name>.bundle, run git clone --mirror <bundle> <workspace>/.saphan/bridge/repos/<name>. A repository with no bundle in the manifest had no refs: create it with git init --bare.
4. Unpack bridge/tx.tar and bridge/documents.tar inside <workspace>/.saphan/bridge/.
5. For each couchdb/<database>.jsonl, create the database on the new node and post its lines to /<database>/_bulk_docs as {"docs":[…],"new_edits":false}, which keeps each document's revision. ⚠ This step is not exercised by the drill.
6. Start the services, then run saphan fleet health and a normal read.

Run the drill on the same backup first. A restore you have not rehearsed is the one that fails on the day.

When it refuses