Title: How to get NixOS hosted at OpenBSD Amsterdam
       Author: Solène
       Date: 07 August 2022
       Tags: nixos openbsd hosting
       Description: This article explains how to host a NixOS VM at the
       hosting company OpenBSD Amsterdam which only provides OpenBSD systems.
       
       # Introduction
       
       In this guide, I'll explain how to create a NixOS VM in the hosting
       company OpenBSD Amsterdam which only provides OpenBSD VMs hosted on
       OpenBSD.
       
       I'd like to thank the team at OpenBSD Amsterdam who offered me a VM for
       this experiment.  While they don't support NixOS officially, they are
       open to have customers running non-OpenBSD systems on their VMs.
       
 (HTM) OpenBSD Amsterdam hosting service website
       
       # The steps from OpenBSD to NixOS
       
       Here is a short description of the steps required to get NixOS
       installed on OpenBSD Amsterdam.
       
       1. Generate a NixOS VM disk file or use the one I provide
       2. Rent a VM at OpenBSD Amsterdam (5€ / month for 1 vCPU, 1GB of
       memory and 50 GB of hdd, with a dedicated IPv4, working IPv6 and
       reverse DNS)
       3. Connect to the hypervisor in order to get the serial console access
       to your VM
       4. Connect with ssh to your VM to reboot it
       5. In the serial console, upon reboot, boot on bsd.rd (the OpenBSD
       installer ramdisk)
       6. Overwrite the local disk by fetching your NixOS VM disk file through
       http/ftp and writing it on the disk file
       7. Reboot on NixOS
       8. Configure the network from the serial console, rebuild the system
       9. Enjoy
       
       # How to proceed
       
       You need to order a VM at OpenBSD Amsterdam first.  You will receive an
       email with your VM name, its network configuration (IPv4 and IPv6), and
       explanations to connect to the hypervisor.  We will need to connect to
       the hypervisor to have a serial console access to the virtual machine. 
       A serial console is a text interface to a machine, you get the machine
       output displayed in your serial console client, and what you type is
       sent to the machine as if you had a keyboard connected to it.
       
       It can be useful to read the onboarding guide before starting.
       
 (HTM) OpenBSD Amsterdam onboarding guide
       
       ## Get into the OpenBSD installer
       
       Our first step is to get into the OpenBSD installer, so we can use it
       to overwrite the disk with our VM.
       
       Connect to the hypervisor, attach to your virtual machine serial
       console by using the following command, we admit your VM name is "vm40"
       in the example:
       
       ```shell
       vmctl console vm40
       ```
       
       You can leave the console anytime by typing "~~." to get back into your
       ssh shell.  The keys sequence "~." is used to drop ssh or a local
       serial console, but when you need to leave a serial console from a ssh
       shell, you need to use "~~.".
       
       You shouldn't see anything because you won't get anything displayed
       until something is showed in the machine virtual first tty, you can
       press "enter" and you should see a login prompt.  We don't need it, but
       it confirms the serial console is working.
       
       In parallel, connect to your VM using ssh, find the root password at
       the end of ~/.ssh/authorized_keys, use "su -" to become root and run
       "reboot".
       
       You should see the shutdown sequence scrolling in the hypervisor ssh
       session displaying the serial console, wait for the machine to reboot
       to spot for the login prompt, in which you will type bsd.rd:
       
       ```text
       Using drive 0, partition 3.
       Loading......
       probing: pc0 com0 mem[638K 3838M 4352M a20=on]
       disk: hd0+
       >> OpenBSD/amd64 BOOT 3.53
       com0: 115200 baud
       switching console to com0
       >> OpenBSD/amd64 BOOT 3.53
       boot> bsd.rd [ENTER] # you need to type bsd.rd
       ```
       
       ## Copy the NixOS VM from the installer
       
       In this step, we will use the installer to fetch the NixOS VM disk and
       overwrite the local disk with it.
       
       * in the installer, type "S" to get a shell:
       
       ```text
       [...]
       Welcome to the OpenBSD/amd64 7.2 installation program.
       (I)nstall, (U)pgrade, (A)utoinstall or (S)hell?
       ```
       
       * enable the network using DHCP with the command:
       
       ```shell
       ifconfig vio0 up autoconf
       ```
       
       * create the disk device in /dev because it's missing by default:
       
       ```shell
       cd /dev
       sh MAKEDEV sd0
       ```
       
       * fetch the NixOS disk and overwrite the local drive with it:
       * (remove the gunzip part if you didn't compress your VM disk file)
       
       ```
       ftp -o - https://perso.pw/nixos/vm.disk.gz | gunzip -f -c | dd of=/dev/rsd0c bs=10M
       ```
       
       * reboot using the command "reboot"
       
       ## NixOS grub menu
       
       At this step, in the serial console you should see a GRUB boot menu, it
       will boot the first entry after a few seconds.  Then NixOS will start
       booting.  In this menu you can access older versions of your system.
       
       After the text stopped scrolling press enter.  You should see a login
       prompt, you can log in with the username "root" and the default
       password "nixos" if you used my disk image.
       
       ## Configuring NixOS
       
       If you used my template, your VM still doesn't have network
       connectivity, you need to edit the file /etc/nixos/configuration.nix in
       which I've put the most important variables you want to customize at
       the top of the file.  You need to configure your IPv4 and IPv6
       addresses and their gateways, and also your username with an ssh key to
       connect to it, and the system name.
       
       Once you are done, run "nixos-rebuild switch", you should have network
       if you configured it correctly.
       
       After the rebuild, run "passwd your_user" if you want to assign a
       password to your newly declared user.
       
       You should be able to connect to your VM using its public IP and your
       ssh key with your username.
       
       EXTRA: You may want to remove the profile minimal.nix which is
       imported: it disables documentation and the use of X libraries, but
       this may trigger packages compilation as they are not always built
       without X support.
       
       ## Resizing the partition (last step)
       
       Because we started with a small 2 GB raw disk to create the virtual
       machine, the partition still has 2 GB only.  We will have to resize the
       partition /dev/vda1 to take all the disk space, and then resize the
       ext4 file system.
       
       First step is to extend the partition to 50 GB, the size of the virtual
       disk offered at openbsd.amsterdam.
       
       ```shell
       # nix-shell -p parted
       # parted /dev/vda
       (parted) resizepart 1
       Warning: The partition /dev/vda1 is currently in use. Are you sure to continue?
       Yes/No? yes
       End? [2147MB]? 50GB
       (parted) quit
       ```
       
       Second step is to resize the file system to fill up the partition:
       
       ```shell
       # resize2fs /dev/vda1
       The file system /dev/vda1 is mounted on / ; Resizing done on the fly
       old_desc_blocks = 1, new_desc_blocks = 6
       The file system /dev/vda1 now has a size of 12206775 blocks (4k).
       ```
       
       Done! "df -h /" should report the new size.
       
       ## Congratulations
       
       You have a fully functional NixOS VM!
       
       # Creating the VM
       
       While I provide a bootable NixOS disk image at
       https://perso.pw/nixos/vm.disk.gz , you can generate yours with this
       guide.
       
       * create a raw disk of 2 GB to install the VM in it
       
       ```shell
       qemu-img create -f raw vm.disk 2G
       ```
       
       * run qemu in a serial console to ensure it works, in the grub boot
       menu you will need to select the 4th choice enabling serial console in
       the installer.  In this no graphics qemu mode, you can stop qemu by
       pressing "ctrl+a" and then "c" to drop into qemu's own console, and
       type "quit" to stop the process.
       
       ```shell
       qemu-system-x86_64 \
         -smp 2 -m 4G \
         -enable-kvm \
         -display curses -nographic \
         -cdrom nixos-minimal*.iso \
         -drive file=vm.disk,if=virtio,format=raw
       ```
       
       * we create the partitions and prepare the chroot
       
       ```
       sudo -i
       parted /dev/vda -- mklabel msdos
       parted /dev/vda -- mkpart primary 1MiB 100%
       mkfs.ext4 -L nixos /dev/vda1
       mount /dev/disk/by-label/nixos /mnt
       mkdir -p /mnt/etc/nixos/
       ```
       
       * edit the file /mnt/etc/nixos/configuration.nix , the NixOS install
       has nano available by default, but you can have your favorite editor by
       using "nix-shell -p vim" if you prefer vim.  Here is a configuration
       file that will work:
       
 (DIR) NixOS configuration.nix file for OpenBSD Amsterdam
       
       * edit the file /mnt/etc/nixos/hardware-configuration.nix
       
 (DIR) NixOS hardware-configuration.nix file for OpenBSD Amsterdam
       
       * we can run the installer, it will ask for the root password, and then
       we can shut down the VM
       
       ```shell
       nixos-install
       systemctl poweroff
       ```
       
       Now, you have to host the disk file somewhere to make it available
       through http or ftp protocol in order to retrieve it from the
       openbsd.amsterdam VM.  I'd recommend compressing the file by running
       gzip on it, that will drastically reduce its size from 2GB to ~500MB.
       
       # Full disk encryption
       
       The ext4 file system offers a way to encrypt specific directories, it
       can be enough for most users.
       
       However, if you want to enable full disk encryption, you need to use
       the guide above to generate your VM, but you need to create a separate
       /boot partition and create a LUKS volume for the root partition.  This
       is explained in the NixOS manual, in the installer section.  You should
       adapt the according bits in the configuration file to match your new
       setup.
       
       Don't forget you will need to connect to the hypervisor to type your
       password through the serial access every time you will reboot.
       
       # Known issue and workaround
       
       There is an issue with the OpenBSD hypervisor and Linux kernels at the
       moment, when you reboot your Linux VM, the VM process on the OpenBSD
       host crashes.  Fortunately, it crashes after all the shutdown process
       is done, so it doesn't let the file system in a weird state.
       
       This problem is fixed in OpenBSD -current as of August 2022, and won't
       happen in OpenBSD 7.2 hypervisors that will be available by the end of
       the year.
       
       A simple workaround is to open a tmux session in the hypervisor to run
       an infinite loop regularly checking if your VM is running, and starting
       it when it's stopped:
       
       ```shell
       while true ; do vmctl status vm40 | grep stopped && vmctl start vm40 ; sleep 30 ; done
       ```
       
 (HTM) Mailing list archives: vmx_fault_page: uvm_fault returns 14, GPA=0xfe001818, rip=0xffffffffc0d6bb96
 (HTM) Mailing list archives: vmm page fault with VM upgraded from Ubuntu 18LTS to 20LTS
       
       # Conclusion
       
       It's great to have more choice when you need a VM.  The OpenBSD
       Amsterdam team is very kind, professional and regularly give money to
       the OpenBSD project.
       
       # Going further
       
       This method should work for other hosting providers, given you can
       access the VM disk from an live environment (installer, rescue system
       etc..).  You may need to pay attention to the disk device, and if you
       can't obtain a serial console access to your system, you need to get
       the network done right in the VM before copying it to the disk.
       
       In the same vein, you can use this method to install any operating
       system supported by the hypervisor.  I chose NixOS because I love this
       system, and it's easy to reproduce a result with its declarative
       paradigm.