Automatic backups with restic and systemd services

Restic is a mature, effective and secure backup solution that covers a wide range of use cases. I use it heavily for all of my backups. The program itself is peak open source: it’s free (as in BSD-2-Clause license), runs on any modern OS, runs on systems low on resources and it can be learned in 30 minutes by reading the well-written documentation. It provides almost everything you need. Except for one thing, which we’re looking at today: Fully automated backups using a systemd service and timer.

I’m too lazy to run restic on a regular basis myself on my servers. Over the last years I iterated several times over my workflow and I think it reached an acceptable level of maturity. It started with simple cron jobs, later systemd units which ran bash scripts and now I’m down to a systemd-only service unit and a timer. This triggers backups on well defined intervals, and creates new snapshots in a fully automated way. If the systemd timer/service fails I get notifications via my existing monitoring utility.

Threat model considerations

Before we start with the actual automation, an important security caveat: I store the backup password in plain text on the server system. The file can only be read by root. This means an attacker who can obtain root rights will be able to access the restic backups, including the snapshot history. In my threat scenario an attacker with root rights does not obtain anything moreby being able to access the backup, so this is fine.

Consider your own threat model if this is fine for you. In most practical cases I think it’s reasonable safe to assume that an attacker with root rights does not get anything more than they already have, when they can access to the backup system, given they already have root access to the underlying machine.

Pre-requirements

This automation only covers the creation of new snapshots. You will need to have the backup repository initialized yourself before you can start.

You can do this yourself using the following, or (even better) consult the quick start guide.

$ restic -r REPOSITORY init

I won’t cover the details here, as the documentation does a much better job already.

When using ssh (sftp in the repository argument), you likely want to also setup passwordless ssh key authentication to the host you’re pushing the data to. Please ensure this is done in a safe way, e.g. you never login as root onto the target machine for pushing backups, ecc.

Automating restic with systemd services

We store the restic secrets and configuration in an environment file, which can only be read by root, e.g. /etc/.restic.env:

RESTIC_REPOSITORY=<sftp:USER@REMOTE:PATH>
RESTIC_PASSWORD=<redacted>
RESTIC_COMPRESSION=<auto|max|none>

Replace the variables above to your liking. We need the repository, the password and (optionally) the compression. The file should have 0400 mode and be owned by root, so that the secrets are not exposed to anyone else.

Then we create the systemd unit and timer files, e.g. /etc/systemd/system/restic-backup.service and /etc/systemd/system/restic-backup.timer. Those files can have 0644 mode as the secrets ares stored elsewhere. In my case I create the backup and also remove old snapshots according to a retention policy in the same service unit. I also perform a backup check at the end to ensure the repository is in a healthy state, however both steps are optional.

# /etc/systemd/system/restic-backup.service

[Unit]
Description=Generate a restic backup snapshot
After=network.target

[Service]
Type=oneshot
WorkingDirectory=/root
ExecStart=/usr/bin/restic backup /etc /root /srv /home /var/log --tag auto
ExecStart=/usr/bin/restic forget --prune --keep-hourly 6 --keep-daily 7 --keep-weekly 4 --keep-monthly 6 --tag auto
ExecStart=/usr/bin/restic check --read-data
EnvironmentFile=/etc/.restic.env
Environment=XDG_CACHE_HOME=/var/cache
Nice=19
IOSchedulingClass=best-effort 
IOSchedulingPriority=7
TimeoutSec=3600
Restart=no

[Install]
WantedBy=multi-user.target

I also add Nice and IOSchedulingPriority to the restic commands, so that the backup runs in the background. This gives priority to other system services, both in terms of CPU and in terms of IO load, so that creating a backup will not have that much of an impact on the performance of running services.

Adapt the retention policy in the restic forget command to your liking. I also mark the automatic backups with the auto tag, which is optional, but I do like it.

And the corresponding timer unit:

# /etc/systemd/system/restic-backup.timer

[Unit]
Description=Daily backups

[Timer]
OnCalendar=daily
Persistent=true

[Install]
WantedBy=timers.target

After creating those, you need to reload the systemd daemon and enable the timer. Then check the service by manually running the restic-backup.service once, and see if everything works as expected. To do so, follow the following procedure:

# systemctl daemon-reload
# systemctl enable --now restic-backup.timer
# systemctl start restic-backup.service
# journalctl -eu restic-backup.service

That’s it. Happy backup-ing!

Additional notes

  • Replace Type=oneshot with Type=simple to not block on the start operation.
  • The initial service run can take some time. Be patient.
  • Systemd services require a full path. I use openSUSE, so the paths are made for that. You might need to adapt it to your needs.
  • Don’t forget to monitor your backups externally. Do not rely on silent automation!
  • No backups without restore test.