What you will set up
A backup on the same server does not survive losing the server. Most of our guides end with "copy the backup to another machine": this guide shows how, for any Debian 13 server.
restic makes encrypted, deduplicated backups: each night it sends only what changed, and every backup, called a snapshot, can be restored on its own. The backups go to a second server, for example a small second Melonslab server, which runs restic's REST server. That server runs in append-only mode: the server you back up can add new snapshots, but it cannot delete or change old ones. If someone takes over that server, or ransomware encrypts it, last week's backups are still there. Old snapshots are removed by the backup server itself, on a schedule you set.
restic and its REST server are open-source projects, under the BSD 2-Clause licence, run by volunteers on GitHub. restic was started in 2014 by Alexander Neumann, and its main developers give Germany as their location. restic sends no telemetry: during a backup it connected only to the repository it was given. We used the Debian packages, which update with the rest of the system.
Every step below was run on two fresh Melonslab VC-P Alloy servers (2 vCPU, 8 GB) with Debian 13:
- The first server backed up
/etc,/root,/home,/srv,/usr/localand/var/libevery night to the second, over TLS with a pinned certificate, plus dumps of PostgreSQL, of MariaDB and of a PostgreSQL in a rootless Podman container. - Deleting a snapshot from the first server failed with
403 Forbidden, and the snapshots were all still there. - The backup server kept 7 daily, 4 weekly and 6 monthly snapshots and removed the rest, out of 76 that covered 210 days.
- A single file, a whole home directory and all three databases were restored, and matched the originals.
restic checkfound no errors. - A failed backup showed a warning at the next login, and both servers ran their timers again after a reboot.
The first backup of a small server took 3 seconds, and its 65 MB took 14 MB on the backup server after deduplication and compression. restic used about 100 MB of memory while it ran, and the REST server about 20 MB.
Before you start
You need:
- the server you want to back up, with Debian 13, set up as in the security guide;
- a second server with Debian 13 for the backups, set up the same way, with enough disk for them. It should be a different machine from the first: a second Melonslab server works.
The examples use 203.0.113.10 and 2001:db8::10 for the server you back up, which this guide calls web1, and 203.0.113.20 and 2001:db8::20 for the backup server. Replace them throughout. Commands run as root, and each step says on which server.
1. Set up the backup server
On the backup server, install the REST server, restic itself (for removing old snapshots in step 8) and htpasswd:
apt update
apt install -y restic-rest-server restic apache2-utils
The package creates a user of its own, restic-rest-server, without a login, and the server runs as that user. Create the folder for the backups:
mkdir -p /srv/restic
chown restic-rest-server: /srv/restic
chmod 700 /srv/restic
The backups travel over the internet, so the REST server speaks TLS. It needs a certificate, and a certificate from Let's Encrypt needs a DNS name. Instead, make your own for the server's addresses, valid for 10 years. The server you back up trusts this one certificate and nothing else, which is stricter than a public certificate:
openssl req -x509 -newkey ec -pkeyopt ec_paramgen_curve:prime256v1 -nodes -days 3650 \
-subj "/CN=backup" -addext "subjectAltName=IP:203.0.113.20,IP:2001:db8::20" \
-keyout /etc/restic-rest-server/tls.key -out /etc/restic-rest-server/tls.crt
chgrp restic-rest-server /etc/restic-rest-server/tls.key
chmod 640 /etc/restic-rest-server/tls.key
Each server you back up gets its own user and password on the backup server. Make a long password, and add the user web1 with it:
openssl rand -base64 32
htpasswd -B /etc/restic-rest-server/users.htpasswd web1
htpasswd asks for the password twice. Keep it at hand for step 3. Now replace the contents of /etc/default/restic-rest-server with:
# Listen on port 8000, IPv4 and IPv6.
LISTEN = :8000
# One repository per server you back up, under this folder.
BACKUP_DIR = /srv/restic
# --append-only: clients can add backups, but never delete or change them.
# --private-repos: each user can only reach the repository with its own name.
ARGS = "\
--htpasswd-file /etc/restic-rest-server/users.htpasswd \
--append-only \
--private-repos \
--tls \
--tls-cert /etc/restic-rest-server/tls.crt \
--tls-key /etc/restic-rest-server/tls.key \
"
With --private-repos, the user web1 can only use /srv/restic/web1, so a second server you back up cannot read or fill the first one's repository. Start the server:
systemctl enable --now restic-rest-server
journalctl -u restic-rest-server -o cat
The log ends with start server on [::]:8000 and TLS enabled. Open the port only to the servers you back up, with both of their addresses. We did not have ufw on our backup server, so these two rules are untested:
ufw allow from 203.0.113.10 to any port 8000 proto tcp
ufw allow from 2001:db8::10 to any port 8000 proto tcp
Finally, print the certificate, and copy it with the BEGIN and END lines:
cat /etc/restic-rest-server/tls.crt
2. Let the two servers reach each other
If both servers are at Melonslab, in the same network, they cannot reach each other until their traffic goes through the gateway: set that up on both servers first, as in the guide to connecting two servers. A backup server elsewhere needs no change. Either way, check from this server that the port answers, over each address you plan to use:
timeout 5 bash -c '</dev/tcp/2001:db8::20/8000' && echo open
3. Install restic and create the repository
The rest of the guide runs on the server you back up, unless a step says otherwise. Debian 13 has restic 0.18.0, which has everything this guide uses:
apt install -y restic
mkdir -p /etc/restic
chmod 700 /etc/restic
Paste the certificate from step 1 into /etc/restic/server.crt. Then make the repository's password. restic encrypts everything with it before it leaves the server, so the backup server never sees your files:
openssl rand -base64 32 > /etc/restic/password
chmod 600 /etc/restic/password
cat /etc/restic/password
Store this password somewhere other than the server too, in your password manager. Without it, nobody can restore the backups: not you, not us. If the server is lost, so is the copy in /etc/restic.
Create /etc/restic/env, with the user's password from step 1 in place of REST_PASSWORD, and lock it:
RESTIC_REPOSITORY=rest:https://[2001:db8::20]:8000/web1/
RESTIC_REST_USERNAME=web1
RESTIC_REST_PASSWORD=REST_PASSWORD
RESTIC_PASSWORD_FILE=/etc/restic/password
RESTIC_CACERT=/etc/restic/server.crt
RESTIC_CACHE_DIR=/var/cache/restic
chmod 600 /etc/restic/env
The last part of the address, web1/, must be the user's name. IPv4 works too, as rest:https://203.0.113.20:8000/web1/: the certificate covers both addresses. Load the settings into your shell, and create the repository:
set -a; . /etc/restic/env; set +a
restic init
created restic repository 4e462201c7 at rest:https://[2001:db8::20]:8000/web1/
Run the set -a line again in each new shell before you use restic by hand.
4. Choose what to back up
For a server set up as in our guides, these folders hold everything you would miss:
/etc: the system's settings, ufw's rules and the server's SSH keys./rootand/home: with rootless Podman, as in the Podman guide, each app's user keeps its settings, its Quadlet files, its volumes under~/.local/share/containers/storage/volumesand its~/backupfolder in its home directory./srv,/usr/localand/var/lib, where other programs keep their data, and/var/spool/cron, for crontabs.
Some things are better left out. Create /etc/restic/excludes:
# Package lists, downloaded again by apt update.
/var/lib/apt/lists
# Database files change while they are copied: back up a dump instead.
/var/lib/postgresql
/var/lib/mysql
# Container image layers, downloaded again by podman pull.
/home/*/.local/share/containers/storage/overlay
# Caches.
/root/.cache
/home/*/.cache
Image layers are the largest part of a Podman user's storage, 293 MB for one PostgreSQL image on our server, and come back with podman pull. Volumes stay in the backup.
Files of a running database are not a safe backup: they change while restic reads them, and the copy may not be usable. That is why the database folders are left out, and step 6 backs up a dump instead. Volumes that hold an app's database are copied too, but restore the app from its dump, or from the ~/backup file its own guide makes.
5. Run it every night
Create /etc/systemd/system/restic-backup.service:
[Unit]
Description=Back up this server with restic
Wants=network-online.target
After=network-online.target
OnFailure=restic-backup-failed.service
[Service]
Type=oneshot
EnvironmentFile=/etc/restic/env
Nice=10
IOSchedulingClass=idle
# Files and folders.
ExecStart=/usr/bin/restic backup --tag files --exclude-file /etc/restic/excludes --exclude-caches /etc /root /home /srv /usr/local /var/lib /var/spool/cron
# After a backup that worked, remove the warning from one that failed.
ExecStartPost=/usr/bin/rm -f /etc/motd.d/restic-backup
--exclude-caches also leaves out every folder that marks itself as a cache. If a backup fails, OnFailure starts a second service, which leaves a warning that you see at your next login. Create /etc/systemd/system/restic-backup-failed.service:
[Unit]
Description=Warn at login that the restic backup failed
[Service]
Type=oneshot
ExecStart=/bin/sh -c "mkdir -p /etc/motd.d && echo \"The backup failed on $$(date). See: journalctl -u restic-backup\" > /etc/motd.d/restic-backup"
And the timer, /etc/systemd/system/restic-backup.timer:
[Unit]
Description=Back up this server every night
[Timer]
OnCalendar=*-*-* 02:00
RandomizedDelaySec=1h
Persistent=true
[Install]
WantedBy=timers.target
The backup starts at a random time between 02:00 and 03:00, so that several servers do not all start at once, and Persistent=true runs a missed backup as soon as the server is back on. Run the first backup now, and switch on the timer:
systemctl daemon-reload
systemctl start restic-backup
journalctl -u restic-backup -o cat
systemctl enable --now restic-backup.timer
Files: 4008 new, 0 changed, 0 unmodified
Dirs: 356 new, 0 changed, 0 unmodified
Added to the repository: 47.438 MiB (9.488 MiB stored)
processed 4008 files, 65.225 MiB in 0:03
snapshot afedc961 saved
systemctl list-timers restic-backup.timer shows the next run. To check on the backups at any time:
systemctl --failed
journalctl -u restic-backup --since yesterday
restic snapshots
6. Back up databases
restic can run a command and store what it prints as a file in a snapshot. If the command fails, so does the backup, with no half-finished dump stored. Add one ExecStart line per database to the service from step 5, before the ExecStartPost line, and run systemctl daemon-reload.
For PostgreSQL from Debian's package:
ExecStart=/usr/bin/restic backup --tag db --stdin-filename postgresql.sql --stdin-from-command -- runuser -u postgres -- pg_dumpall
For MariaDB from Debian's package:
ExecStart=/usr/bin/restic backup --tag db --stdin-filename mariadb.sql --stdin-from-command -- mariadb-dump --all-databases --single-transaction --routines --events
For PostgreSQL in a rootless Podman container, as in our app guides, the dump runs as the app's user. Here the user is notes, the container notes-db, and its database and database user are both notes:
ExecStart=/usr/bin/restic backup --tag db --stdin-filename notes-db.sql --stdin-from-command -- systemd-run -M notes@ --user -P -q podman exec notes-db pg_dump -U notes notes
systemd-run -M notes@ --user runs the command in that user's session, where its containers are. runuser and sudo -u are not enough: Podman then fails with OCI permission denied.
Each dump becomes a snapshot of its own, tagged db. On our server, with the container stopped, the backup failed, systemctl --failed listed restic-backup.service, and the next login showed:
The backup failed on Fri Oct 2 06:47:39 PM CEST 2026. See: journalctl -u restic-backup
7. Check that the server cannot delete its backups
Try to remove all but the last snapshot:
restic forget --keep-last 1 --prune
Remove(<snapshot/afedc96152>) failed: unexpected HTTP response (403): 403 Forbidden
unable to remove snapshot/afedc961524b... from the repository
...
3 snapshots have been removed, running prune
...
Fatal: unexpected HTTP response (403): 403 Forbidden
Despite the line in the middle, nothing was removed: restic snapshots lists them all. The REST server refuses to delete or overwrite anything, so neither you nor an intruder can remove them from here. Removing a single snapshot with restic forget and its ID also fails with 403 Forbidden, but restic 0.18.0 exits with status 0 there, so do not trust its exit code. The failed prune leaves a few duplicate index entries, which restic check calls non-critical. The prune in step 8 tidies them up.
8. Remove old backups on the backup server
Snapshots are removed by the backup server, which reads the repository directly from disk. It needs the repository's password for that, so the server that keeps the backups can also read them. Both servers are yours, and the server you back up still cannot delete anything.
On the backup server, store the password from step 3 where only root can read it:
mkdir -p /etc/restic
chmod 700 /etc/restic
nano /etc/restic/web1.password
chmod 600 /etc/restic/web1.password
Create /etc/systemd/system/restic-prune@.service, one service for all the servers you back up:
[Unit]
Description=Remove old restic snapshots of %i
[Service]
Type=oneshot
User=restic-rest-server
Group=restic-rest-server
UMask=027
# The password is readable only by root. systemd hands this service a copy.
LoadCredential=password:/etc/restic/%i.password
CacheDirectory=restic-prune
Environment=RESTIC_REPOSITORY=/srv/restic/%i RESTIC_PASSWORD_FILE=%d/password RESTIC_CACHE_DIR=/var/cache/restic-prune
ExecStart=/usr/bin/restic forget --prune --retry-lock 2h --keep-daily 7 --keep-weekly 4 --keep-monthly 6
It runs as the REST server's user, so the files it writes keep the right owner, and LoadCredential gives it the password without letting that user read /etc/restic. The policy keeps the last snapshot of each of the last 7 days, 4 weeks and 6 months, for the files and for each database separately. Change the numbers to suit you. --retry-lock 2h waits if a backup is running.
And /etc/systemd/system/restic-prune@.timer:
[Unit]
Description=Remove old restic snapshots of %i every day
[Timer]
OnCalendar=*-*-* 12:00
RandomizedDelaySec=1h
Persistent=true
[Install]
WantedBy=timers.target
Switch it on for web1, and run it once:
systemctl daemon-reload
systemctl enable --now restic-prune@web1.timer
systemctl start restic-prune@web1
journalctl -u restic-prune@web1 -o cat
Our test repository had 76 snapshots of the files, one every three days over 210 days, made with restic's --time option. The service kept 12 of them, and removed the data only the others used:
Applying Policy: keep 7 daily, 4 weekly, 6 monthly snapshots
...
60 snapshots have been removed, running prune
...
done
For another server you back up, add a user for it in step 1, its password in /etc/restic/<name>.password, and enable restic-prune@<name>.timer.
9. Restore
Run these on the server you back up, with the settings loaded as in step 3. latest means the newest snapshot, and since each database has snapshots of its own, pick the files with --tag files. To see what is there:
restic snapshots
restic ls latest --tag files /etc/ssh
To restore a single file to a new folder, so nothing is overwritten:
restic restore latest --tag files --target /root/restore --include /etc/ssh/sshd_config
It lands in /root/restore/etc/ssh/sshd_config. To restore a whole folder to a new path, such as an app user's home directory:
restic restore latest:/home/notes --tag files --target /root/notes-restore
The files keep their owners and permissions. A Podman volume's files are in .local/share/containers/storage/volumes/<volume>/_data. Copy back what you need, and remove the restored folder afterwards.
To restore a database, stream its dump back in. PostgreSQL from Debian's package:
restic dump latest --tag db --path /postgresql.sql /postgresql.sql | runuser -u postgres -- psql -X
pg_dumpall restores every database and role. Into a server that already has some of them, it prints errors such as role "postgres" already exists for those, and restores the rest: drop a damaged database first with dropdb. MariaDB:
restic dump latest --tag db --path /mariadb.sql /mariadb.sql | mariadb
PostgreSQL in the rootless container:
restic dump latest --tag db --path /notes-db.sql /notes-db.sql | systemd-run -M notes@ --user -P -q podman exec -i notes-db psql -X -U notes notes
To restore onto a new server after losing the old one, install restic there, recreate /etc/restic from step 3 with the password from your password manager, and restore as above.
10. Test your restores
A backup you have never restored is a hope, not a backup. Once a month, restore one file and one database dump on the server, and check that they are what you expect. Also let restic check the repository, reading a tenth of the stored data each time:
restic check --read-data-subset 10%
read 10.0% of data packs
no errors were found
restic check without the option checks the structure only, and reads no data.
Troubleshooting
connect: no route to host or TLS handshake timeout between two Melonslab servers. Their traffic does not go through the gateway. Set up the guide to connecting two servers on both servers.
x509: certificate signed by unknown authority. restic does not use your certificate: RESTIC_CACERT is missing from /etc/restic/env, or the settings are not loaded in this shell.
unexpected HTTP response (401): 401 Unauthorized. The user name or its password is wrong, or the address does not end with the user's own name: with --private-repos, web1 can only use /web1/.
Fatal: wrong password or no key found. /etc/restic/password does not hold the repository's password from step 3. Perhaps you pasted the user's password from step 1 there instead.