Title: Using dpb on OpenBSD for package compilation cluster
       Author: Solène
       Date: 30 May 2021
       Tags: openbsd
       Description: 
       
       # Introduction
       
       Today I will explain how to easily setup your own OpenBSD dpb infra. 
       dpb is a tool to manage port building and can use chroot to use a sane
       environment for building packages.
       
       This is particularly useful when you want to test packages or build
       your own, it can parallelize package compilation in two way: multiples
       packages at once and multiples processes for one package.
       
 (HTM) dpb man page
 (HTM) proot man page
       
       The dpb and proot executable files are available under the bin
       directory of the ports tree.
       
       Building your packages provide absolutely NOTHING compared to using
       binary packages except wasting CPU time, disk space and bandwidth.
       
       # Setup
       
       You need a ports tree and a partition that you accept to mount with
       wxallowed,nosuid,dev options.  I use /home/ for that.  To simplify the
       setup, we will create a chroot in /home/build/ and put our ports tree
       in /home/build/usr/ports (then your /usr/ports can be a symlink).  
       
       Create a text file that will be used as a configuration file for proot
       
       ```proot configuration file
       chroot=/home/build
       WRKOBJDIR=/tmp/pobj
       LOCKDIR=/tmp/locks
       PLIST_REPOSITORY=/data/plist
       DISTDIR=/data/distfiles
       PACKAGE_REPOSITORY=/data/packages
       actions=unpopulate
       sets=base comp etc xbase xfont xshare xetc xserver
       ```
       
       This will tell proot to create a chroot in /home/build and preconfigure
       some variables for /etc/mk.conf, use all sets listed in "sets" and
       clean everything when run (this is what actions=unpopulate is doing). 
       Running proot is as easy as "proot -c proot_config".
       
       Then, you should be able to run "dpb -B /home/build/ some/port" and it
       will work.
       
       # Ease of use
       
       I wrote a script to clean locks from dpb, locks from ports system and
       pobj directories but also taking care of adding the mount options.
       
       Options -p and -j will tell dpb how many cores can be used for parallel
       compilation, note that dpb is smart and if you tell it 3 ports in
       parallel and 3 threads in parallel, it won't use 3x3, it will compile
       three ports at a time and once it's stuck with only one port, it will
       add cores to its build to make it faster.
       
       ```script shell
       #!/bin/sh
       
       CHROOT=/home/build/
       CORES=3
       
       rm -fr ${CHROOT}/usr/ports/logs/amd64/locks/*
       rm -fr ${CHROOT}/tmp/locks/*
       rm -fr ${CHROOT}/tmp/pobj/*
       mount -o dev -u /home
       mount -o nosuid -u /home
       mount -o wxallowed -u /home
       /usr/ports/infrastructure/bin/dpb -B $CHROOT -c -p $CORES -j $CORES  $*
       ```
       
       Then I use "doas ./my_dpb.sh sysutils/p5-Rex lang/guile" to run the
       build process.
       
       It's important to use -c in dpb command line which will clear
       compilation logs of the packages but retains the log size, this will be
       used to estimate further builds progress by comparing current log size
       with previous logs sizes.
       
       You can harvest your packages from /home/build/data/packages/ , I even
       use a symlink from /usr/ports/packages/ to the dpb packages directory
       because sometimes I use make in ports and sometimes I use dpb, this
       allow recompiling packages in both areas.  I do the same for distfiles.
       
       # Going further
       
       dpb can spread the compilation load over remote hosts (or even manage
       compilation for a different architecture), it's not complicated to
       setup but it's out of scope for the current guide.  This requires
       setting up ssh keys and NFS shares, the difficulty is to think with the
       correct paths depending on chroot/not chroot and local / nfs.
       
       I extremely recommend reading dpb man pages, it supports many options
       such as providing it a list of pkgpaths (package address such as
       editor/vim or www/nginx) or building ports in random order.
       
       Here is a simply command to generate a list of pkgpaths of outdated
       packages on your system compared to the ports tree, the -q parameter is
       to make it a lot quicker but less accurate for shared libraries.
       
       ```command line example
       /usr/ports/infrastructure/bin/pkg_outdated -q | awk '/\// { print $1 }'
       ```
       
       # Conclusion
       
       I use dpb when I want to update my packages from sources because the
       binary packages are not yet available or if I want to build a new
       package in a clean environment to check for missing dependencies,
       however I use a simple "make" when I work on a port.