Linux Backup and Recovery with restic and a Restore Drill
You will build an encrypted, deduplicated backup with restic and prove it works with an actual restore. By the end you will follow the 3-2-1 rule and have a verified recovery path, not just hope.
Learning Objectives
- Initialize an encrypted restic repository.
- Take scheduled, deduplicated snapshots.
- Restore data and verify integrity.
- Time: ~3 hours · Difficulty: Intermediate · Prereqs: a server plus an offsite/secondary backup target.
Architecture Overview
Environment Setup
You will need: restic installed and a backup target (local disk plus an offsite location such as object storage).
Before you begin: store the repository password in a secret manager, losing it means losing access to backups.
Prerequisites
- Linux with restic installed (
sudo apt install -y restic) and a backup target disk or path. - Root or sudo access for reading /etc and /srv.
- A password file for the repository encryption key, kept outside the backup.
- Decide what to back up (/etc and /srv used here) and a retention goal.
Step-by-Step Execution
restic encrypts and deduplicates at the repo level, so backups are private and space-efficient.
openssl rand -base64 32 | tr -d '\n' > /root/.restic-pass && chmod 600 /root/.restic-pass && export RESTIC_PASSWORD_FILE=/root/.restic-pass && restic -r /backup/repo initExpected output:
created restic repository 9c2a1f3e at /backup/repo
Please note that knowledge of your password is required to access the repository.What this teaches: restic init creates the repo structure; all data is encrypted client-side with the password, so the storage backend never sees plaintext.
Session note: the password file must exist before init, and every restic command in steps 02 to 05 reads RESTIC_PASSWORD_FILE from the environment. Run those commands in this same shell, or add export RESTIC_PASSWORD_FILE=/root/.restic-pass to /root/.bashrc so future shells and scheduled jobs authenticate. Mode 600 keeps the file readable by root alone; keep it outside every backup set. Alternatively, pass --password-file /root/.restic-pass before the subcommand on each invocation.
restic -r /backup/repo backup /etc /srv --tag daily
Expected output:
Files: 142 new, 0 changed, 0 unmodified
Dirs: 18 new, 0 changed, 0 unmodified
snapshot 3f8b2d1a savedWhat this teaches: Snapshots are deduplicated and tagged; repeated backups transfer only changed blocks, making daily snapshots cheap.
Pruning keeps storage bounded while retaining enough history to recover.
restic -r /backup/repo forget --keep-daily 7 --keep-weekly 4 --prune
Expected output:
Applying Policy: keep 7 daily, 4 weekly snapshots
keep 11 snapshots:
remove 2 snapshots:
prune: removing obsolete packsWhat this teaches: forget applies the retention policy (--keep-daily/weekly) and --prune reclaims space; without pruning, storage grows forever.
Expected output:
What this teaches: Restoring to a separate target with --include proves the backup is readable; untested backups are just hope with extra steps.
One local repository is only one copy. The 3-2-1 rule needs a second repository on different media, with one copy offsite. Initialize a second repo on the offsite target and copy every snapshot into it. In this lab a mounted disk at /mnt/offsite stands in for the offsite target, but a second folder on the same physical disk does not satisfy 3-2-1: in production the second repo lives on genuinely separate media or another site, for example restic -r sftp:backup@backup-host:/srv/restic/repo init followed by the same copy --from-repo invocation. Both repos share the same password file in this lab, which is why the copy command passes --from-password-file for the source repo explicitly.
INIT THE OFFSITE REPO AND COPY SNAPSHOTS
mkdir -p /mnt/offsite && restic -r /mnt/offsite/repo init && restic -r /mnt/offsite/repo copy --from-repo /backup/repo --from-password-file /root/.restic-passExpected output:
snapshot 3f8b2d1a of (1/1) savedWhat this teaches: restic copy moves snapshots between repositories without re-reading source data. The offsite repo is now an independent, encrypted second copy, which is what makes the 3-2-1 claim in this lab real. Repeat the copy, or schedule it, after each backup so the offsite copy stays current.
Confirm the copy landed in the second repository:
restic -r /mnt/offsite/repo snapshots
Expected output:
ID Time Host Tags Paths
---------------------------------------------------------------
3f8b2d1a 2026-09-29 10:15:02 server daily /etc /srv
1 snapshotsWhat this teaches: restic snapshots lists what a repository actually holds; checking the offsite repo closes the loop on the second copy.
How It Works
restic uses content-defined chunking: files are split into variable-size blocks, each hashed, and only new hashes are stored. Identical blocks across files and snapshots are stored once, which is why daily snapshots of /etc cost almost nothing after the first. Encryption is applied before anything leaves the host, so untrusted storage (S3, SFTP, a USB disk) is safe.
Retention is a policy, not an accident. --keep-daily 7 --keep-weekly 4 declares how much history matters; forget selects which snapshots to drop and --prune repacks the repo to reclaim the space. Without this loop, every backup system eventually fills its disk and fails at the worst moment.
The restore drill is the actual deliverable. Backups fail in boring ways (wrong include paths, expired credentials, corrupted repos), and only a rehearsed restore to an alternate target proves recoverability. Scheduling the drill is as important as scheduling the backup.
Progress So Far
Testing & Validation
restic -r /backup/repo check && diff /etc/hostname /tmp/restore/etc/hostnameThe repository check should report no errors and the diff should be empty. If both pass, your backups are recoverable, not just present.
restic check validates the repository structure and indexes, which is fast but does not re-read every data blob. For a stronger and slower verification, run restic -r /backup/repo check --read-data periodically: it reads all pack files and catches bit rot that a metadata-only check misses. On large repos, check --read-data-subset 10% verifies a different tenth of the data on each run.
Troubleshooting
| Symptom | Likely cause | Fix |
|---|---|---|
| restic: wrong password | RESTIC_PASSWORD_FILE points at the wrong file or changed | Verify the file path and contents; there is no recovery without the password |
| Backup extremely slow on first run | Initial snapshot reads everything | Expected once; later runs are incremental. Exclude caches and tmp |
| Prune runs out of memory | Prune loads indexes into RAM and repacks data; very large repos can exhaust memory | Prune more often with smaller policies, cap repack batch size with --max-repack-size, or run prune on a host with more RAM. Note: --max-unused sets how much unused data is tolerated before repacking; it is not a memory limit |
| Snapshot missing expected files | Include paths wrong or permission denied during backup | Check the backup log for errors and verify paths with restic ls |
| Repo lock contention | Concurrent restic processes on one repo | Serialize with flock, or use restic unlock after a crashed run |
| Restore target permission errors | Running restore as the wrong user | Match ownership with the original, or restore as root and chown after |
Troubleshooting
- Wrong password: restic cannot recover without the repo password; store it safely and redundantly.
- Repo locked: a previous run was interrupted; run
restic unlock. - Slow first backup: initial snapshots are full; later ones dedupe and are fast.
Extension Ideas
- Schedule backups with a systemd timer.
- Copy to offsite object storage to complete the 3-2-1 rule.
- Compare with database-aware backups in DB Backup & Recovery.
Key Results
- Created encrypted, deduplicated backups with restic.
- Proved recovery with a real restore drill, not assumptions.
- Bounded storage with a 7-daily/4-weekly retention policy.
- Verified repository integrity with
restic check.
Key Takeaways
- Encrypt client-side; never trust the storage backend.
- Deduplication makes frequent snapshots cheap.
- Retention plus prune keeps storage bounded.
- A restore drill is the only proof a backup works.
- Exclude caches and transient data from every backup set.
- Restore to a scratch target first, never over the live system.
- Keep one copy offsite; the 3-2-1 rule still applies.
Practice Challenges
Challenge 1: 3-2-1 drill
Goal: back up /etc to two different backends (local disk and SFTP), then restore one file from each and diff against the original.
Hint: restic -r selects the repo per command.
Challenge 2: Retention math
Goal: configure keep-daily/weekly/monthly, run forget --dry-run, and explain which snapshots would be dropped and why.
Hint: --dry-run plus -v shows the selection.