Title: Making Qubes OS backups more efficient
Author: Solène
Date: 12 May 2023
Tags: qubes qubesos backup
Description: In this article, you will learn how to send Qubes OS
backups to restic or borg backup software tools in order to make your
backups more efficient.
# Introduction
These days, I've been playing a lot with Qubes OS, it has an
interesting concept of deploying VMs (using Xen) in a well integrated
and transparent manner in order to hardly separate every tasks you
need.
By default, you get default environments such as Personal, Work and an
offline Vault, plus specials VMs to handle USB proxy, network and
firewall. What is cool here is that when you run a program from a VM,
only the window is displayed in your window manager (xfce), and not the
whole VM desktop.
The cool factor with this project is their take on the real world
privacy and security need, allowing users to run what they need to run
(proprietary software, random binaries), but still protect them. Its
goal is totally different from OpenBSD and Tails. Did I say you can
also route a VM network through Tor out of the box? =D
If you want to learn more, you can visit Qubes OS website (or ask if
you want me to write about it):
(HTM) Qubes OS official website
(HTM) New user guide: How to organize your cubes (nice reading to understand Qubes OS)
# Backups
If you know me, you should know I'm really serious about backups. This
is incredibly important to have backups.
Qubes OS has a backup tool that can be used out of the box, it just
dump the VMs storage into an encrypted file, it's easy but not
efficient or practical enough for me.
If you want to learn more about the format used by Qubes OS (and how to
open them outside of Qubes OS), they wrote some documentation:
(HTM) Qubes OS: backup emergency restore
Now, let's see how to store the backups in Restic or Borg in order to
have proper backups.
/!\ While both software support deduplication, this doesn't work well
in this case because the stored data are compressed + encrypted
already, which has a very high entropy (it's hard to find duplicated
patterns).
# Backup tool
Qubes OS backup tool offers compression and encryption out of the box,
but when it comes to the storage location, we can actually use a
command to send the backups to the command's stdin, and guess what,
both restic and borg support receiving data on their standard input!
I'll demonstrate how to proceed both with restic and borg with a simple
example, I recommend to build your own solution on top of it the way
you need.
(IMG) Screenshot of Qubes backup tool
# Create a backup VM
As we are running Qubes OS, I prefer to create a dedicated backup VM
using the Fedora template, it will contain the passphrase to the
repository and an SSH key for remote backup.
You need to install restic/borg in the template to make it available in
that VM.
If you don't know how to install software in a template, it's well
documented:
(HTM) Qubes OS: how to install software
Generate an SSH key if you want to store your data on a remote server
using SSH, and deploy it on the remote server.
# Write a backup script
In order to simplify the backup command configuration in the backup
tool (it's a single input line), but don't sacrifice on features like
pruning, we will write a script on the backup VM doing everything we
need.
While I'm using a remote repository in the example, nothing prevents
you from using a local/external drive for your backups!
The script usage will be simple enough for most tasks:
* `./script init` to create the repository
* `./script backup` to create the backup
* `./script list` to display snapshots
* `./script restore $snapshotID` to restore a backup, the output file
will always be named `stdin`
## Restic
Write a script in `/home/user/restic.sh` in the backup VM, it will
allow simple customization of the backup process.
```shell
#!/bin/sh
export RESTIC_PASSWORD=mysecretpass
# double // is important to make the path absolute
export RESTIC_REPOSITORY=sftp://solene@10.42.42.150://var/backups/restic_qubes
KEEP_HOURLY=1
KEEP_DAYS=5
KEEP_WEEKS=1
KEEP_MONTHS=1
KEEP_YEARS=0
case "$1" in
init)
restic init
;;
list)
restic snapshots
;;
restore)
restic restore --target . $2
;;
backup)
cat | restic backup --stdin
restic forget \
--keep-hourly $KEEP_HOURLY \
--keep-daily $KEEP_DAYS \
--keep-weekly $KEEP_WEEKS \
--keep-monthly $KEEP_MONTHS \
--keep-yearly $KEEP_YEARS \
--prune
;;
esac
```
Obviously, you have to change the password, you can even store it in
another file and use the according restic option to load the passphrase
from a file (or from a command). Although, Qubes OS backup tool
enforces you to encrypt the backup (which will be store in restic), so
encrypting the restic repository won't add any more security, but it
can add privacy by hiding what's in the repo.
/!\ You need to run the script with the parameter "init" the first
time, in order to create the repository:
```shell
$ chmod +x restic.sh
$ ./restic.sh init
```
## Borg
Write a script in `/home/user/borg.sh` in the backup VM, it will allow
simple customisation of the backup process.
```shell
#!/bin/sh
export BORG_PASSPHRASE=mysecretpass
export BORG_REPO=ssh://solene@10.42.42.150/var/solene/borg_qubes
KEEP_HOURLY=1
KEEP_DAYS=5
KEEP_WEEKS=1
KEEP_MONTHS=1
KEEP_YEARS=0
case "$1" in
init)
borg init --encryption=repokey
;;
list)
borg list
;;
restore)
borg extract ::$2
;;
backup)
cat | borg create ::{now} -
borg prune \
--keep-hourly $KEEP_HOURLY \
--keep-daily $KEEP_DAYS \
--keep-weekly $KEEP_WEEKS \
--keep-monthly $KEEP_MONTHS \
--keep-yearly $KEEP_YEARS
;;
esac
```
Same explanation as with restic, you can save the password elsewhere or
get it from a command, but Qubes backup already encrypt the data, so
the repo encryption will mostly only add privacy.
/!\ You need to run the script with the parameter "init" the first
time, in order to create the repository:
```shell
$ chmod +x borg.sh
$ ./borg.sh init
```
## Configure Qubes backup
Now, configure the Qubes backup tool:
* Choose the VMs to backup
* Check "Compress backups", because it's done before encryption it
yields a better efficiency than compression done by restic on the
encrypted data
* Click Next
* Choose the backup VM in the "Target qube" list
* In the field "backup directory or command" type `/home/user/restic.sh
backup` or `/home/user/borg.sh backup` depending on your choice
* Pick a passphrase
* Run the backup
# Restoring a backup
While it's nice to have backups, it's important to know how to use
them. The setup doesn't add much complexity, and the helper script
will ease your life.
On the backup VM, run `./borg.sh list` (or the restic version) to
display available snapshots in the repository, then use `./borg.sh
restore $snap` with the second parameter being a snapshot identifier
listed in the earlier command.
You will obtain a file named `stdin`, this is the file to use in Qubes
OS restore tool.
# Warning
If you don't always backup all the VMs, if you keep the retention
policy like in the example above, you may lose data.
For example, if you have a KEEP_HOURLY=1, create a backup of all your
VMs, and just after, you specifically want to backup a single VM, you
will lose the previous full backup due to the retention policy.
In some cases, it may be better to not have any retention policy, or
simply time based (keep snapshots which date < n days).
# Conclusion
Using this configuration, you get all the features of a industry
standard backup solution such as integrity check, retention policy or
remote encrypted storage.
# Troubleshoot
In case of an issue with the backup command, Qubes backup will display
a popup message with the command output, this helps a lot debugging
problems.
An easy way to check if the script works by hand is to run it from the
backup VM:
```shell
echo test | ./restic.sh backup
```
This will create a new backup with the data "test" (and prune older
backups, so take care!), if it doesn't work this is a simple way to
trigger a new backup to solve your issue.