Title: Creating a NixOS live USB for a full featured APU router
Author: Solène
Date: 03 August 2022
Tags: networking security nixos apu
Description: This explains how to create a live USB image of NixOS to
boot on an APU router including all required features for your network.
# Introduction
At home, I'm running my own router to manage Internet, run DHCP, do
filter and caching etc... I'm using an APU2 running OpenBSD, it works
great so far, but I was curious to know if I could manage to run NixOS
on it without having to deal with serial console and installation.
It turned out it's possible! By configuring and creating a live NixOS
USB image, one can plug the USB memory stick into the router and have
an immutable NixOS.
(HTM) NixOS wiki about creating a NixOS live CD/USB
# Network diagram
Here is a diagram of my network. It's really simple except the bridge
part that require an explanation. The APU router has 3 network
interfaces and I only need 2 of them (one for WAN and one for LAN), but
my switch doesn't have enough ports for all the devices, just missing
one, so I use the extra port of the APU to connect that device to the
whole LAN by bridging the two network interfaces.
```
+----------------+
| INTERNET |
+----------------+
|
|
|
+----------------+
| ISP ROUTER |
+----------------+
| 192.168.1.254
|
|
| 192.168.1.111
+----------------+
| APU ROUTER |
+----------------+
|bridge #2 and #3|
| 10.42.42.42 |
+----------------+
|port #3 |
| | port #2
+----------+ |
| |
| +--------+ +----------+
| 10.42.42.150 | switch |-----| Devices |
+--------+ +--------+ +----------+
| NAS |
+--------+
```
Here is a list of services I need on my router, this doesn't include
all my filtering rules and specific tweaks.
- DHCP server
- DNS resolving caching using unbound
- NAT
- SSH
- UPnP
- Munin
- Bridge ethernets ports #2 and #3 to use #3 as an extra port like a
switch
# The whole configuration
For the curious, here is the whole configuration of the setup. In the
sections after, I'll explain each parts of the code.
```nix
{ config, pkgs, ... }:
{
isoImage.squashfsCompression = "zstd -Xcompression-level 5";
powerManagement.cpuFreqGovernor = "ondemand";
boot.kernelPackages = pkgs.linuxPackages_xanmod_latest;
boot.kernelParams = [ "copytoram" ];
boot.supportedFilesystems = pkgs.lib.mkForce [ "btrfs" "vfat" "xfs" "ntfs" "cifs" ];
services.irqbalance.enable = true;
networking.hostName = "kikimora";
networking.dhcpcd.enable = false;
networking.usePredictableInterfaceNames = true;
networking.firewall.interfaces.eth0.allowedTCPPorts = [ 4949 ];
networking.firewall.interfaces.br0.allowedTCPPorts = [ 53 ];
networking.firewall.interfaces.br0.allowedUDPPorts = [ 53 ];
security.sudo.wheelNeedsPassword = false;
services.acpid.enable = true;
services.openssh.enable = true;
services.unbound = {
enable = true;
settings = {
server = {
interface = [ "127.0.0.1" "10.42.42.42" ];
access-control = [
"0.0.0.0/0 refuse"
"127.0.0.0/8 allow"
"10.42.42.0/24 allow"
];
};
};
};
services.miniupnpd = {
enable = true;
externalInterface = "eth0";
internalIPs = [ "br0" ];
};
services.munin-node = {
enable = true;
extraConfig = ''
allow ^63\.12\.23\.38$
'';
};
networking = {
defaultGateway = { address = "192.168.1.254"; interface = "eth0"; };
interfaces.eth0 = {
ipv4.addresses = [
{ address = "192.168.1.111"; prefixLength = 24; }
];
};
interfaces.br0 = {
ipv4.addresses = [
{ address = "10.42.42.42"; prefixLength = 24; }
];
};
bridges.br0 = {
interfaces = [ "eth1" "eth2" ];
};
nat.enable = true;
nat.externalInterface = "eth0";
nat.internalInterfaces = [ "br0" ];
};
services.dhcpd4 = {
enable = true;
extraConfig = ''
option subnet-mask 255.255.255.0;
option routers 10.42.42.42;
option domain-name-servers 10.42.42.42, 9.9.9.9;
subnet 10.42.42.0 netmask 255.255.255.0 {
range 10.42.42.100 10.42.42.199;
}
'';
interfaces = [ "br0" ];
};
time.timeZone = "Europe/Paris";
users.mutableUsers = false;
users.users.solene.initialHashedPassword = "$6$ffffffffffffffff$TTTTTTTTTTTTTTTTTTTTTTTTTTTTTTTTTTTTTT.aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa";
users.users.solene = {
isNormalUser = true;
extraGroups = [ "sudo" "wheel" ];
};
}
```
# Explanations
This setup deserves some explanations with regard to each part of it.
## Live USB specific
I prefer to use zstd instead of xz for compressing the liveUSB image,
it's way faster and the compression ratio is nearly identical as xz.
```nix
isoImage.squashfsCompression = "zstd -Xcompression-level 5";
```
There is currently an issue when trying to use a non default kernel,
ZFS support is pulled in and create errors. By redefining the list of
supported file systems you can exclude ZFS from the list.
```nix
boot.supportedFilesystems = pkgs.lib.mkForce [ "btrfs" "vfat" "xfs" "ntfs" "cifs" ];
```
## Kernel and system
The CPU frequency should stay at the minimum until the router has some
load to compute.
```nix
powerManagement.cpuFreqGovernor = "ondemand";
services.acpid.enable = true;
```
This makes the system to use the XanMod Linux kernel, it's a set of
patches reducing latency and improving performance.
(HTM) Xanmod XanMod project website
```nix
boot.kernelPackages = pkgs.linuxPackages_xanmod_latest;
```
In order to reduce usage of the USB memory stick, upon boot all the
content of the liveUSB will be loaded in memory, the USB memory stick
can be removed because it's not useful anymore.
```nix
boot.kernelParams = [ "copytoram" ];
```
The service irqbalance is useful as it assigns certain IRQ calls to
specific CPUs instead of letting the first CPU core to handle
everything. This is supposed to increase performance by hitting CPU
cache more often.
```nix
services.irqbalance.enable = true;
```
## Network interfaces
As my APU wasn't running Linux, I couldn't know the name if the
interfaces without booting some Linux on it, attach to the serial
console and check their names. By using this setting, Ethernet
interfaces are named "eth0", "eth1" and "eth2".
```nix
networking.usePredictableInterfaceNames = true;
```
Now, the most important part of the router setup, doing all the
following operations:
- assign an IP for eth0 and a default gateway
- create a bridge br0 with eth1 and eth2 and assign an IP to br0
- enable NAT for br0 interface to reach the Internet through eth0
```nix
networking = {
defaultGateway = { address = "192.168.1.254"; interface = "eth0"; };
interfaces.eth0 = {
ipv4.addresses = [
{ address = "192.168.1.111"; prefixLength = 24; }
];
};
interfaces.br0 = {
ipv4.addresses = [
{ address = "10.42.42.42"; prefixLength = 24; }
];
};
bridges.br0 = {
interfaces = [ "eth1" "eth2" ];
};
nat.enable = true;
nat.externalInterface = "eth0";
nat.internalInterfaces = [ "br0" ];
};
```
This creates a user solene with a predefined password, add it to the
wheel and sudo groups in order to use sudo. Another setting allows
wheel members to run sudo without password, this is useful for testing
purpose but should be avoided on production systems. You could add
your SSH public key to ease and secure SSH access.
```nix
users.mutableUsers = false;
security.sudo.wheelNeedsPassword = false;
users.users.solene.initialHashedPassword = "$6$bVPyGA3aTEMTIGaX$FYkFnOqwk8GNfeLEfppgGjZ867XxirQ19v1337.GSRdzxw7JrRi6IcpaEdeSuNTHSxIIhunter2Iy6clqB14b0";
users.users.solene = {
isNormalUser = true;
extraGroups = [ "sudo" "wheel" ];
};
```
## Networking services
This will run a DHCP server advertising the local DNS server and the
default gateway, as it defines ranges for DHCP clients in our local
network.
```nix
services.dhcpd4 = {
enable = true;
extraConfig = ''
option subnet-mask 255.255.255.0;
option routers 10.42.42.42;
option domain-name-servers 10.42.42.42, 9.9.9.9;
subnet 10.42.42.0 netmask 255.255.255.0 {
range 10.42.42.100 10.42.42.199;
}
'';
interfaces = [ "br0" ];
};
```
All systems require a name in order to work, and we don't want to use
DHCP to get the IPs addresses. We also have to define a time zone.
```nix
networking.hostName = "kikimora";
networking.dhcpcd.enable = false;
time.timeZone = "Europe/Paris";
```
This enables OpenSSH daemon listening on port 22.
```nix
services.openssh.enable = true;
```
This enables the service unbound, a DNS resolver that is able to do
some caching as well. We need to allow our network 10.42.42.0/24 and
listen on the LAN facing interface to make it work, and not forget to
open the ports TCP/53 and UDP/53 in the firewall. This caching is very
effective on a LAN server.
```nix
services.unbound = {
enable = true;
settings = {
server = {
interface = [ "127.0.0.1" "10.42.42.42" ];
access-control = [
"0.0.0.0/0 refuse"
"127.0.0.0/8 allow"
"10.42.42.0/24 allow"
];
};
};
};
networking.firewall.interfaces.br0.allowedTCPPorts = [ 53 ];
networking.firewall.interfaces.br0.allowedUDPPorts = [ 53 ];
```
This enables the service miniupnpd, this can be quite dangerous because
its purpose is to allow computer on the network to create NAT
forwarding rules on demand. Unfortunately, this is required to play
some video games and I don't really enjoy creating all the rules for
all the video games requiring it.
```nix
services.miniupnpd = {
enable = true;
externalInterface = "eth0";
internalIPs = [ "br0" ];
};
```
This enables the service munin-node and allow a remote server to
connect to it. This service is used to gather metrics of various data
and make graphs from them. I like it because the agent running on the
systems is very simple and easy to extend with plugins, and on the
server side, it doesn't need a lot of resources. As munin-node listens
on the port TCP/4949 we need to open it.
```nix
services.munin-node = {
enable = true;
extraConfig = ''
allow ^13\.17\.23\.28$
'';
};
networking.firewall.interfaces.eth0.allowedTCPPorts = [ 4949 ];
```
# Conclusion
By building a NixOS live image using Nix, I can easily try a new
configuration without modifying my router storage, but I could also use
it to ssh into the live system to install NixOS without having to deal
with the serial console.