Jails with nullfs mount on FreeBSD 10 without buildworld -------------------------------------------------------- Last edited: $Date: 2010/08/10 21:01:58 $ > FreeBSD jails are one ot the features that make FreeBSD awesome. Jails are mostly used for enhanced security but they help whenever working in silos offers an advantage. Very often one is adviced not to mix packages build through the ports system with binary packages installed by pkg. With jails it is easy to keep them separated. Also jails can be useful in testing some application or some setup. Et ceterea, there can be many more reasons to setup jails. This is a setup without ezjail. Of course working with ezjail is simple but setting up everything by hand helps to learn a little. The system I did this on does not have a lot of computing power so I choose for a way to do this without building world. The use of nullfs mounts (comparable with bind mounts on Linux) saves diskspace and because that part is mounted readonly in the jail adds some security. I started with a fresh install of FreeBSD 10 i386. Most of the stuff below I did according to the very good FreeBSD handbook and some other recources I found on the internet. ## Install cpdup If you have a fresh FreeBSD10 system, you may need to install cpdup. pkg install cpdup ## General overview The jail system to create has the following layout: home |-- j | |-- mroot | |-- skel | |-- myjail | `-- mysecondjail |-- js | |-- myjail | `-- mysecondjail * /home/j/mroot: this holds the base system that will be mounted readonly over nullfs * /home/j/skel: this is a template for the read-write part of the jail * /home/j/myjail: this is where the base system gets mounted for the jail 'myjail' * /home/js/myjail: this is where the read-write part of the jail lives The directory /home/j/mroot holds the base system. For every jail this will be mounted readonly over nullfs. So this part will only exists once on your harddisk. The /home/j/skel contains the variable files that differs from jail to jail, like /etc/rc.conf. Setting up the directory for a new jail consists of two parts: * creating a directory in /home/j/ in which the base system will be mounted readonly * copy the /home/j/skel to a directory in /home/js/, this will holds the variable files like /etc/rc.conf of the jail and the home directories of the users. Also it contains jail specific stuff, like jail specific applications. Also, a new jail requires adding two lines to /etc/fstab, adding an instance to /etc/jail.conf and some finetuning. ## Prepare base system The base system will be mounted read-only into the jail. We populate the base system from the install CDROM, so we don't have to do a buildworld. mkdir -p /home/j/mroot setenv DESTDIR /home/j/mroot mount_cd9660 /dev/cd0 /mnt tar -xf /mnt/usr/freebsd-dist/base.txz -C $DESTDIR tar -xf /mnt/usr/freebsd-dist/doc.txz -C $DESTDIR tar -xf /mnt/usr/freebsd-dist/ports.txz -C $DESTDIR cd umount /mnt cp /etc/resolv.conf $DESTDIR/etc/ chroot $DESTDIR tzsetup pkg install cpdup With this last step (pkg install cpdup) we force the system to install the pkg utility. Later on in the process this will not be as easy as this, because we are going to move directories like /etc out of the base system. ## Prepare template for the read-write part of the jail First setup the directory for the template: mkdir /home/j/skel /home/j/skel/home /home/j/skel/usr-X11R6 /home/j/skel/distfiles Now populate it by moving parts of the base system into it: cd /home/j/mroot mv etc /home/j/skel mv usr/local /home/j/skel/usr-local mv tmp /home/j/skel mv var /home/j/skel mv root /home/j/skel And create symlinks back into the base system: cd /home/j/mroot mkdir s ln -s s/etc etc ln -s s/home home ln -s s/root root ln -s /s/usr-local usr/local ln -s /s/usr-X11R6 usr/X11R6 ln -s /s/distfiles usr/ports/distfiles ln -s s/tmp tmp ln -s s/var var Setup make.conf: echo "WRKDIRPREFIX?= /s/portbuild" >> /home/j/skel/etc/make.conf ## Creating the directories for a new jail Two directories are created for the new jail: * /home/j/myjail * /home/js/myjail The /home/j/myjail will be populated by a nullfs mount, the /home/js/myjail will get real files by copying the skel directory to it. ### Populate /home/js/myjail cpdup /home/j/skel /home/js/myjail ### Edit /etc/fstab On the host add the following lines to /etc/fstab: /home/j/mroot /home/j/myjail nullfs ro 0 0 /home/js/myjail /home/j/myjail/s nullfs rw 0 0 ### Edit /etc/jail.conf For each jail a block has to be added to /etc/jail.conf: imjail { host.hostname = "myjail"; path = "/home/j/myjail"; ip4.addr += "192.168.1.100/32"; devfs_ruleset = 4; allow.raw_sockets = 0; exec.clean; exec.system_user = "root"; exec.jail_user = "root"; exec.start = "/bin/sh /etc/rc"; exec.stop = "/bin/sh /etc/rc.shutdown"; exec.consolelog = "/var/log/jail_myjail_console.log"; mount.devfs; allow.set_hostname = 0; allow.sysvipc = 0; } ### devfs and FreeBSD 10 The release of FreeBSD 10 brought some challenges around devs. I have not found a great solution for this on the internet. The best is to manually mount devfs into each jail directory before starting the jail services. mount -t devfs /dev /home/j/myjail/dev This can for example be done from /etc/rc.local ### Last steps Mount the new directories for the jail, setup root password, add a user, set the hostname and make sure sshd will only listen to the ip address of the jail: mount -a mount chroot /home/j/myjail passwd adduser vi /etc/rc.conf vi /etc/ssh/sshd_config exit ## Start the jail services jail start myjail ## Mount jail on a remote NFS share Currently I am experimenting with mounting the read write part of the jail from a remote NFS server. My NFS server is a small system running Debian. First, add some lines to /etc/rc.conf: rpcbind_enable="YES" rpc_statd_enable="YES" rpc_lockd_enable="YES" To be sure, I did a reboot of the FreeBSD system, maybe just a restart of rpc.bind is enough. After this, I mounted a remote NFS directory to /home/jnfs, followed by: mkdir /home/j/nfsjail cpdup /home/j/skel /home/jnfs/nfsjail mount_nullfs -o ro /home/j/mroot /home/j/nfsjail/ mount_nullfs -o rw /home/jnfs/nfsjail /home/j/nfsjail/s/ mount -t devfs devfs /home/j/nfsjail/dev/ From there on, editted sshd_config in the nfsjail directory to only listen to the ip address of the jail, set a hostname in the rc.conf and started the jail. Entered the jail with jexec [jailnumber] /bin/sh. The ultimate test was to set the root password and add a user. Both were possible without complaints about not being able to lock the password file. Now that this seems to work, I will try to run a few jails this way and see if this can run stable. The idea behing this is to be able to run a FreeBSD jail server diskless without having to run the jails from iSCSlI. See Building a diskless FreeBSD jail server, gopher://box.matto.nl/0/freebsd10_disklessjailserver.txt. ## Resources There is a lot of good and usefull information on the internet. The following resources were very helpfull: * Creating and Controlling Jails chapter in FreeBSD handbook http://www.freebsd.org/doc/en_US.ISO8859-1/books/handbook/jails-build.html * Updating Multiple Jails chapter in FreeBSD handbook http://www.freebsd.org/doc/handbook/jails-application.html * Multiple FreeBSD Jails with nullfs from http://www.scottro.net/ http://www.scottro.net/nullfsjail.html#FreeBSD-10 $Id: freebsd10jails.txt,v 1.3 2010/08/10 21:01:58 matto Exp $