Title: Introduction to immutable Linux systems
Author: Solène
Date: 12 July 2023
Tags: immutability linux
Description: In this article, you will learn what stands behind the
label immutable when it comes to Linux systems
# Introduction
If you reach this page, you may be interested into this new category of
Linux distributions labeled "immutable".
In this category, one can find by age (oldest → youngest) NixOS,
Guix, Endless OS, Fedora Silverblue, OpenSUSE MicroOS, Vanilla OS and
many new to come.
I will give examples of immutability implementation, then detail my
thoughts about immutability, and why I think this naming can be
misleading. I spent a few months running all of those distributions on
my main computers (NAS, Gaming, laptop, workstation) to be able to
write this text.
# What's immutability?
The word immutability itself refers to an object that can't change.
However, when it comes to an immutable operating system, the definition
immediately become vague. What would be an operating system that can't
change? What would you be supposed to do with it?
We could say that a Linux LIVE-CD is immutable, because every time you
boot it, you get the exact same programs running, and you can't change
anything as the disk media is read only. But while the LIVE-CD is
running, you can make changes to it, you can create files and
directories, install packages, it's not stuck in an immutable state.
Unfortunately, this example was nice but the immutability approach by
those Linux distribution is totally different, so we need to think a
bit further.
There are three common principles in these systems:
* system upgrades aren't done on the live system
* packages changes are applied on the next boot
* you can roll back a change
Depending on the implementation, a system may offer more features. But
this list is what a Linux distribution should have to be labelled
"immutable" at the moment.
# Immutable systems comparison
Now we found what are the minimum requirements to be called immutable,
let's go through each implementation, by their order of appearance.
## NixOS / Guix
In this section, I'm mixing NixOS and Guix as they both rely on the
same implementation. NixOS is based on Nix (first appearance in 2003),
which has been forked into early 2010s into the Guix package manager to
be 100% libre, which gave birth to an eponym operating system also 100%
free.
(HTM) NixOS official project website
(HTM) Guix official project website
(HTM) Jonathan Lorimer's blog post explaining Eelco Dolstra's thesis about Nix
These two systems are really different than a traditional Unix like
system we are used to, and immutability is a main principle. To make
it quick, they are based on their package manager (being Nix or Guix)
that contains every package or built file into a special read-only
directory (where only the package manager can write) where each package
has its own unique entry, and the operating system itself is a
byproduct of the package manager.
What does that imply? If the operating system is built, this is
because it's made of source code, you literally describe what you want
your system to be in a declarative way. You have to list users, their
shells, installed packages, running services and their configurations,
partitions to mount with which options etc... Fortunately, it's made a
lot easier by the use of modules which provide sane defaults, so if you
create a user, you don't have to specify its UID, GID, shell, home
etc...
So, as the system is built and stored in the special read-only
directory, all your system is derived from that (using symbolic links),
so all the files handled by the package manager are read-only. A
concrete example is that /etc/fstab or /bin/sh ARE read-only, if you
want to make a change in those, you have to do it through the package
manager.
I'm not going into details, because this store based package manager is
really different than everything else but:
* you can switch between two configurations on the fly as it's just a
symlink dance to go from a configuration to another
* you can select your configuration at boot time, so you can roll back
to a previous version if something is wrong
* you can't make change to a package file or system file as they are
read only
* the mount points except the special store directory are all mutable,
so you can write changes in /home or /etc or /var etc... You can remove
the system symlinks by a modified version, but you can't modify the
symlink source itself.
This is the immutability as seen through the Nix lens.
I've spent a few years running NixOS systems, this is really a blast
for me, and the best "immutable" implementation around, but
unfortunately it's too different, so its adoption rate is very low,
despite all the benefits.
(HTM) NixOS forum: My issues when pushing NixOS to companies
## Endless OS
While this one is not the oldest immutable OS around, it's the first
one to be released for the average user, while NixOS and Guix are older
but for a niche user category. The company behind Endless OS is trying
to offer a solid and reliable system, free and open source, that can
works without Internet, to be used in countries with a low Internet /
powergrid coverage. They even provide a version with "offline internet
included" containing Wikipedia dumps, class lessons and many things to
make a computer useful while offline (I love their work).
(HTM) Endless OS official project website
Endless OS is based on Debian, but uses the OSTree tool to make it
immutable. OSTree allows you to manage a core system image, and add
layers on top of it, think of packages as layers. But it can also
prepare a new system image for the next boot.
With OSTree, you can apply package changes in a new version of the
system that will be available at next boot, and revert to a previous
version at boot time.
The partitions are mounted writable, except for `/usr`, the land of
packages handled by OSTree, which is mounted read-only. There are no
rollbacks possible for `/etc`.
Programs meant to be for the user (not the packages to be used by the
system like grub, X display or drivers) are installed from Flatpak
(which also uses OSTree, but unrelated to the system), this avoids the
need to reboot each time you install a new package.
My experience with Endless OS is mixed, it is an excellent and solid
operating system, it's working well, never failed, but I'm just not the
target audience. They provide a modified GNOME desktop that looks like
a smartphone menu, because this is what most non-tech users are
comfortable with (but I hate it). And installing DevOps tools isn't
practical but not impossible, so I keep Endless OS for my multimedia
netbook and I really enjoy it.
## Fedora Silverblue
This linux distribution is the long descendant of Project Atomic, an
old initiative to make Fedora / CentOS/ RHEL immutable. It's now part
of the Fedora releases along with Fedora Workstation.
(HTM) Project Atomic website
(HTM) Fedora Silverblue project website
Fedora Silverblue is also using OSTree, but with a twist. It's using
rpm-OSTree, a tool built on top of OSTree to let your RPM packages
apply the changes through OSTree.
The system consists of a single core image for the release, let's say
fedora-38, and for each package installed, a new layer is added on top
of the core. At anytime, you can list all the layers to know what
packages have been installed on top of the core, if you remove a
package, the whole stack is generated again (which is terribly SLOW)
without the package, there is absolutely no leftover after a package
removal.
On boot, you can choose an older version of the system, in case
something broke after an upgrade. If you install a package, you need
to reboot to have it available as the change isn't applied on the
current booted system, however rpm-OSTree received a nice upgrade, you
can temporarily merge the changes of the next boot into the live system
(using a tmpfs overlay) to use the changes.
The mountpount management is a bit different, everything is read-only
except `/etc/`, `/root` and `/var`, but your home directory is by
default in `/var/home` which sometimes breaks expectations. There are
no rollbacks possible for `/etc`.
As installing a new package is slow due to rpm-OSTree and requires a
reboot to be fully usable (the live change back port store the extra
changes in memory), they recommend to use Flatpak for programs, or
`toolbox`, some kind of wrapper that create a rootless fedora container
where you can install packages and use it in your terminal. toolbox is
meant to provide development libraries or tool you wouldn't have in
Flatpak, but that you wouldn't want to install in your base Fedora
system.
(HTM) toolbox website
My experience with Fedora Silverblue has been quite good, it's stable,
the updates are smooth even if they are slow. `toolbox` was working
fine despite I don't find this practical.
## OpenSUSE MicroOS
This spin of OpenSUSE Tumbleweed (rolling-release OpenSUSE) features
immutability, but with its own implementation. The idea of MicroOS is
really simple, the whole system except a few directories like `/home`
or `/var` lives on a btrfs snapshot, if you want to make a change to
the system, the current snapshot is forked into a new snapshot, and the
changes are applied there, ready for the next boot.
(HTM) OpenSUSE MicroOS official project website
What's interesting here is that `/etc` IS part of the snapshots, and
can be roll backed, which wasn't possible in the OSTree based systems.
It's also possible to make changes to any file of the file system (in a
new snapshot, not the live one) using a shell, which can be very
practical for injecting files to solve a driver issue. The downside
it's not guaranteed that your system is "pure" if you start making
changes, because they won't be tracked, the snapshots are just
numbered, and you don't know what changes were made in each of them.
Changes must be done through the command `transactional-update` which
do all the snapshot work for you, and you could either manipulate
package by adding/removing a package, or just start a shell in the new
snapshot to make all the changes you want. I said `/etc` is part of
the snapshots, it's true, but it's never read-only, so you could make a
change live in `/etc`, then create a new snapshot, the change would be
immediately inherited. This can create troubles if you roll back to a
previous state after an upgrade if you also made changes to `/etc` just
before.
The default approach of MicroOS is disturbing at first, a reboot is
planned every day after a system update, this is because it's a
rolling-release system and there are updates every day, and you won't
benefit from them until you reboot. While you can disable this
automatic reboot, it makes sense to use the newest packages anyway, so
it's something to consider if you plan to use MicroOS.
There is currently no way to apply the changes into the live system
(like Silverblue is offering), it's still experimental, but I'm
confident this will be doable soon. As such, it's recommended to use
`distrobox` to use rootless containers of various distributions to
install your favorite tools for your users, instead of using the base
system packages. I don't really like this because this adds
maintenance, and I often had issues of distrobox refusing to start a
container after a reboot, I had to destroy and recreate it entirely to
solve.
(HTM) distrobox GitHub project page
My experience with OpenSUSE MicroOS has been wonderful, it's in
dual-boot with OpenBSD on my main laptop, it's my Linux Gaming OS, and
it's also my NAS operating system, so I don't have to care about
updates. I like that the snapshots system doesn't restrict me, while
OSTree systems just doesn't allow you to make changes without
installing a package.
## Vanilla OS
Finally, the really new (but mature enough to be usable) system in the
immutable family is Vanilla OS based on Ubuntu (but soon on Debian),
using ABroot for immutability. With Vanilla OS, we have another
implementation that really differs from what we saw above.
(HTM) Vanilla OS project website
ABroot named is well thought, the idea is to have a root partition A,
another root partition B, and a partition for persistent data like
`/home` or `/var`.
Here is the boot dance done by ABroot:
* first boot is done on A, it's mounted in read-only
* changes to the system like new packages or file changes in `/etc` are
done on B (and can be applied live using a tmpfs overlay)
* upon reboot, if previous boot was A, you boot on B, then if the boot
is successful, ABroot scan for all the changes between A and B, and
apply all the changes from B to A
* when you are using your system, until you make a change, A and B are
always identical
This implementation has downsides, you can only roll back a change
until you boot on the new version, then the changes are also applied on
the previous boot, and you can't roll back. This implementation mostly
protects you from a failing upgrade, or if you made changes and tried
them live, but you prefer to rollback.
Vanilla OS features the package manager apx, written by distrobox
author. That's for sure an interesting piece of software, allowing
your non-root user to install packages from many distributions (arch
linux, fedora, ubuntu, nix, etc...) and integrates them into the system
as if they were installed locally. I suppose it's some kind of layer
on top of distrobox.
(HTM) apx package manager GitHub project page
My experience wasn't very good, I didn't find ABroot to be really
useful, and the version 22.10 I tried was using an old Ubuntu LTS
release which didn't make my gaming computer really happy. The overall
state of Vanilla OS, ABroot and apx is that they are young, I think it
can become a great distribution, but it still has some rough edges.
## Alpine Linux (with LBU)
I've been told that it was possible to achieve immutability on Alpine
Linux using the "lbu" command.
(HTM) Alpine Linux wiki: Local backup
I don't want to go much into details, but here is the short version:
you can use Alpine Linux installer as a base system to boot from, and
create tarballs of "saved configurations" that are automatically
applied upon boot (it's just tarred directories and some automation to
install packages). At every boot, everything is untarred again, and
packages are installed again (you should use an apk cache directory),
everything in live memory, fully writable.
What does this achieve? You always start from a clean state, changes
are applied on top of it at every boot, you can roll back the changes
and start fresh again. Immutability as we defined above here isn't
achieved because changes are applied on the base system, but it's quite
close to fulfill (my own) requirements.
I've been using it a few days only, not as my main system, and it
requires a very good understanding of what you are doing because the
system is fully in memory, and you need to take care about what you
want to save/restore, which can create big archives.
On top of that, it's poorly documented.
# Pros, Cons and Facts
Now I gave some details about all the major immutable systems (Linux
based) around, I think it's time to list the real pros and cons I found
from my experimentation.
## Pros
* you can roll back changes if something went wrong.
* transactional-updates allows you to keep the system running correctly
during packages changes.
## Cons
* configuration management tool (ansible, salt, puppet etc..) integrate
VERY badly, they received updates to know how to apply package changes,
but you will mostly hit walls if you want to manage those like regular
systems.
* having to reboot after a change is annoying (except for NixOS and
Guix which don't require rebooting for each change).
* OSTree based systems aren't flexible, my netbook requires some extra
files in alsa directories to get sound (fortunately Endless OS have
them!), you just can't add the files without making a package deploying
them.
* blind rollbacks, it's hard to figure what was done in each version of
the system, so when you roll back it's hard to figure what you are
doing exactly.
* it can be hard to install programs like Nix/Guix which require a
directory at the root of the file system, or install non-packaged
software system-wide (this is often bad practice, but sometimes a
necessary evil).
## Facts
* immutability is a lie, many parts of the systems are mutable,
although I don't know how to describe this family with a different word
(transactional something?).
* immutable doesn't imply stateless.
* NixOS / Guix are doing it right in my opinion, you can track your
whole system through a reliable package manager, and you can use a
version control system on the sources, it has the right philosophy from
the ground up.
* immutability is often associated with security benefits, I don't
understand why. If someone obtains root access on your system, they
can still manipulate the live system and have fun with the `/boot`
partition, nothing prevent them to install a backdoor for the next
boot.
* immutability requires discipline and maintenance, because you have to
care about the versioning, you have extra programs like apx / distrobox
/ devbox that must be updated in parallel of the system (while this is
all integrated into NixOS/Guix).
# Conclusion
Immutable operating systems are making the news in our small community
of open source systems, but behind this word lies many implementations
with different use cases. The word immutable certainly creates
expectations from users, but it's really nothing more than
transactional updates for your operating system, and I'm happy we can
have this feature now.
But transactional updates aren't new, I think it started a while ago
with Solaris and ZFS allowing you to select a system snapshot at boot
time, then I'm quite sure FreeBSD implemented this a decade ago, and it
turns out that on any linux distribution with regular btrfs snapshots
you could select a snapshot at boot time.
(HTM) Previous blog post about booting on a BTRFS snapshot without any special setup
In the end, what's REALLY new is the ability to apply a transactional
change on a non-live environment, integrates this into the bootloader,
and give the user the tooling to handle this easily.
# Going further
I recommend reading the blog post "“Immutable” → reprovisionable,
anti-hysteresis" by Colin Walters.
(HTM) “Immutable” → reprovisionable, anti-hysteresis