Backups and restore
Nightly PostgreSQL dumps, where they go and how to test a restore.
All customer data is in PostgreSQL, in three databases:
| Database | Service | Holds |
|---|---|---|
yanu_auth | yanu-auth | Accounts, payments, themes |
yanu_sms | yanu-sms-api | Messages, balances, routes for Malawi HTTP and dashboard sending |
yanu_smpp | yanu-smpp-api | Binds, resellers, prices, top-ups, ledger, SMPP reports |
The backup also keeps the Postgres globals (roles) and the SMPP gateway's saved configuration.
Scripts and schedule
The scripts live in the yanu-infra repo:
| Script | What it does |
|---|---|
scripts/backup.sh | pg_dump -Fc of each database, plus globals and the gateway config, with a SHA256SUMS file. |
scripts/restore.sh test <folder> <database> | Restores into a scratch database, counts the tables, then drops it. Live data is not touched. |
scripts/restore.sh replace <folder> <database> | Replaces the live database. Asks you to type the database name to confirm. |
The systemd timer yanu-backup.timer runs the backup every night.
Where backups go
- On the server in
/var/backups/yanu/<UTC time stamp>/. Kept 14 days (BACKUP_KEEP_DAYS). - Off the server: set
BACKUP_RSYNC_TARGETin/etc/yanu/backup.env, so a lost server does not lose the data. - Encrypted: set
BACKUP_AGE_RECIPIENTin the same file. Dumps are then encrypted withage. Keep the private key off the server, in a safe place. Without it, encrypted backups cannot be restored.
Backups contain personal data and payment records. Limit who can read them.
Test a restore every month
A backup you have never restored is not a backup.
Pick last night's folder, for example ls /var/backups/yanu/.
For each database run, as root on the server, in the yanu-infra folder:
scripts/restore.sh test /var/backups/yanu/<stamp> yanu_auth
scripts/restore.sh test /var/backups/yanu/<stamp> yanu_sms
scripts/restore.sh test /var/backups/yanu/<stamp> yanu_smppFor encrypted backups add AGE_IDENTITY=/path/to/key in front.
Each run checks the checksums and prints restore test ok: <database> has N tables in the backup. A checksum mismatch or a missing dump fails loudly.
Write down the date, the folder used and the result.
Real restore (disaster)
- Stop the service that uses the database, for example
docker compose stop yanu-smpp-api. - Run
scripts/restore.sh replace /var/backups/yanu/<stamp> yanu_smppand type the database name to confirm. - Start the service again and check sign in, sending and balances.
- Tell customers that data after the backup time may be missing. Check top-ups made after it by hand.
- SMPP bind balances live in the gateway, not only in the database. After restoring
yanu_smpp, compare a few binds' balances in the portal with the ledger before you approve new top-ups.