Cohabiting FreeBSD and Gentoo Linux on a Common ZFS Volume

My Librem 15 arrived a while back.  I normally prefer FreeBSD for just about everything, but I need access to a Linux OS running on the Librem platform in order to help me port over some remaining device drivers (namely the BYD mouse and screen brightness).

In order to facilitate this, I’ve installed a setup I developed a while back: that of a dual-booted FreeBSD and Gentoo Linux system living on the same ZFS volume.  This article details how to achieve this setup.

Note that this article is based on the EFI bootloader.  If you insist on legacy BIOS boots, you’ll need to adapt the procedure.

Overview of the Scheme

The scheme is based on a somewhat atypical use of the ZFS filesystem (namely foregoing the mountpoint functionality for the OS datasets in favor of an fstab-based approach) combined with GRUB to achieve a dual-bootable OS

ZFS Overview

The ZFS system differs slightly from the “typical” ZFS setup on both FreeBSD and Linux.  Some datasets (namely the home directories) are shared between both operating systems, but the OS datasets differ in their mount-points depending on which OS we are using, and thus the ZFS-specific mountpoint functionality can’t be effectively used.

In this article, assume that the volume’s name is “data”.

The overall scheme looks something like this:

  • data/home is mounted to /home, with all of its child datasets using the ZFS mountpoint system
  • data/freebsd and its child datasets house the FreeBSD system, and all have their mountpoints set to legacy
  • data/gentoo and its child datasets house the Gentoo system, and have their mountpoints set to legacy as well

OS and GRUB Overview

Both OSes must utilize the /etc/fstab method for mounting most of their filesystems, since we cannot make use of the ZFS mountpoint functionality.  This requires a different fstab for each OS.  Note that the data/home dataset (as well as any other similar dataset) will be mounted using the ZFS mountpoint method, not fstab.

Additionally, both OSes have access to the other OS’ data through a special top-level directory (/freebsd on Gentoo, /gentoo on FreeBSD).

The GRUB bootloader can be used to provide a workable boot selection facility without any serious modification or configuration (other than knowing the magic words to type into the grub.cfg file!)

Setup Procedure

The setup procedure consists of the following steps:

  1. Use the FreeBSD installer to create the GPT and ZFS pool
  2. Install and configure FreeBSD, with the native FreeBSD boot loader
  3. Boot into FreeBSD, create the Gentoo Linux datasets, install GRUB
  4. Boot into the Gentoo Linux installer, install Gentoo
  5. Boot into Gentoo, finish any configuration tasks

Note that nothing stops you from reversing the procedure, installing Gentoo first, and using its tools.  I just find that GPT creation and GRUB installation go a lot more smoothly on FreeBSD.

Getting Ready

To perform the setup procedure, you’ll need installer memstick images for both OSes.  FreeBSD’s installer can be gotten here; Gentoo’s can be gotten here (use the livedvd ISO).  You’ll also need some means of internet access (of course).

Note that in the case of the Librem 15 or similar laptops which have no ethernet connector, you may need to adopt a slightly modified procedure of installing Gentoo’s wireless tools and wpa_supplicant during the main installation.

FreeBSD Installer

Boot into the FreeBSD installer, go through the boot menu, and select manual partitioning mode.  This will drop you into a shell and ask you to create your partitions and mount everything at /mnt.

Create Partitions and ZFS Pool

The first thing to do is use the gpart tool to create your partitions.  FreeBSD’s man pages are rather good, so you can use “man gpart” to get the guide to the tool.  My procedure on the Librem 15 looks like this:

gpart create -s gpt ada0
gpart create -s gpt ada1
gpart add -t efi -l efi-system -s 200M ada0
gpart add -t freebsd-zfs -l zfs-data ada0
gpart add -t linux-swap -l swap -s 96G ada0
gpart add -t freebsd-zfs -l zfs-data-log -s 16G ada0
gpart add -t freebsd-zfs -l zfs-data-cache ada0

Then create a ZFS pool with the new partitions, and format the EFI system partition with the DOS filesystem (seriously, why do we still use that thing?):

newfs_msdos /dev/ada0p1
zpool create -m legacy -o atime=off -o checksum=sha256 data /dev/ada0p2 log /dev/ada0p2 cache /dev/ada0p3

Note that we’ve turned off atime (which reduces disk write traffic considerably) and set the checksum algorithm to sha256.

The ada1 disk is a SSD I had installed.  If you don’t have an SSD, it doesn’t make any sense to have a log or a cache.  The 16GB intent log is way overkill, but it reduces the strain on the device.  Note that we set the root dataset’s mountpoint to “legacy”.

Note that linux has its own swap format, so we can’t share the swap device.

Create the ZFS Datasets

Now that you have a ZFS pool, the first thing you’ll need to do is create the datasets.   Start by creating the FreeBSD root and mounting it (note that it will inherit the “legacy” mountpoint from its parent):

zfs create -o compression=lz4 data/freebsd
mount -t zfs data/freebsd /mnt/

We need to create some mountpoint directories:

mkdir /mnt/home
mkdir /mnt/gentoo/
mkdir /mnt/tmp
mkdir /mnt/usr
mkdir /mnt/var

I use a fairly elaborate ZFS scheme, which applies different executable, setuid, and compression properties for certain directories.  This achieves a significant compression ratio, effectively increasing the size of my disks:

zfs create -o exec=on -o setuid=off -o compression=off data/freebsd/tmp
zfs create -o exec=on -o setuid=on -o compression=lz4 data/freebsd/usr
zfs create -o exec=off -o setuid=off -o compression=gzip data/freebsd/usr/include
zfs create -o exec=on -o setuid=off -o compression=lz4 data/freebsd/usr/lib
zfs create -o exec=on -o setuid=off -o compression=lz4 data/freebsd/usr/lib32
zfs create -o exec=on -o setuid=off -o compression=gzip data/freebsd/usr/libdata
zfs create -o exec=on -o setuid=on -o compression=lz4 data/freebsd/usr/local
zfs create -o exec=on -o setuid=off -o compression=gzip data/freebsd/usr/local/etc
zfs create -o exec=off -o setuid=off -o compression=gzip data/freebsd/usr/local/include
zfs create -o exec=on -o setuid=off -o compression=lz4 data/freebsd/usr/local/lib
zfs create -o exec=on -o setuid=off -o compression=lz4 data/freebsd/usr/local/lib32
zfs create -o exec=on -o setuid=off -o compression=gzip data=freebsd/usr/local/libdata
zfs create -o exec=on -o setuid=off -o compression=gzip data/freebsd/usr/local/share
zfs create -o exec=off -o setuid=off -o compression=off data/freebsd/usr/local/share/info
zfs create -o exec=off -o setuid=off -o compression=off data/freebsd/usr/local/share/man
zfs create -o exec=on setuid=on -o compression=lz4 data/freebsd/obj
zfs create -o exec=on -o setuid=on -o compression=lz4 data/freebsd/usr/ports
zfs create -o exec=off -o setuid=off -o compression=lz4 data/freebsd/usr/ports
zfs create -o exec=on -o setuid=off -o compression=gzip data/freebsd/usr/share
zfs create -o exec=off -o setuid=off -o compression=off data/freebsd/usr/share/info
zfs create -o exec=off -o setuid=off -o compression=off data/freebsd/usr/share/man
zfs create -o exec=off -o setuid=off -o compression=gzip data/freebsd/usr/src
zfs create -o exec=off -o setuid=off -o compression=lz4 data/freebsd/var
zfs create -o exec=off -o setuid=off -o compression=off data/freebsd/var/db
zfs create -o exec=off -o setuid=off -o compression=lz4 data/freebsd/var/db/pkg
zfs create -o exec=off -o setuid=off -o compression=gzip data/freebsd/var/log
zfs create -o exec=off -o setuid=off -o compression=off data/freebsd/var/empty
zfs create -o exec=off -o setuid=off -o compression=gzip data/freebsd/var/mail
zfs create -o exec=on -o setuid=off -o compression=off data/freebsd/var/tmp

Because FreeBSD is pretty strict about where certain files go, this scheme works pretty well.  You could of course continue to subdivide it up to your heart’s desire, creating more subdirectories in /usr/share and such.

For Gentoo, you’ll probably want a simpler scheme, as Linux tends to be much sloppier about the locations of things:

zfs create -o exec=on -o setuid=off -o compression=off data/gentoo/tmp
zfs create -o exec=on -o setuid=on -o compression=lz4 data/gentoo/usr
zfs create -o exec=off -o setuid=off -o compression=lz4 data/gentoo/var

A Gentoo master might be able to subdivide this up as I’ve done with FreeBSD.

The final task is to mount all these filesystems manually with the following command template:

mount -t zfs data/freebsd/<path> /mnt/<path>

This is necessary, as all the mount-points will be “legacy”.  I won’t waste space by showing all the commands here.

Install and Configure the FreeBSD system

Now type “exit”, which will drop you back into the FreeBSD installer, with everything mounted at /mnt/.  The remainder of the installation procedure is straightforward.  However, you’ll need to opt to go to a shell for two final configuration tasks.

Go to the shell, then enter into the new FreeBSD system

chroot /mnt

Create the fstab

Since we mount the vast majority of the ZFS datasets to different paths in each OS, we’ll need to create an /etc/fstab file for them.  The following fstab will mount all the datasets to the right locations:

data/freebsd/tmp /tmp zfs rw 0 0
data/freebsd/usr /usr zfs rw 0 0
data/freebsd/usr/include /usr/include zfs rw 0 0
data/gentoo/ /gentoo zfs rw 0 0
data/gentoo/tmp /gentoo/tmp zfs rw 0 0
proc /proc procfs rw 0 0

Note that I’ve left out all a number of the entries.  You’ll have to map each dataset to its proper path as shown above.

Install the FreeBSD Native Bootloader

We’ll need the FreeBSD bootloader to get into the system for the first time.  Install it to the with the following procedure:

mount -t msdosfs /dev/ada0p1 /mnt
mkdir /mnt/efi
mkdir /mnt/efi/BOOT
cp /boot/boot1.efi /mnt/efi/BOOT/BOOTX64.EFI

The last thing you’ll need to do is set the bootfs parameter on the zpool, so the FreeBSD bootloader will pick the right dataset:

zpool set -o bootfs=data/freebsd data

You may also need to set the bootme flag on the EFI system partition for some hardware:

gpart set -a bootme -i 1 ada0

Your system is now ready to boot into the OS directly.

FreeBSD Main OS

You should now be able to boot into FreeBSD directly.  You’ll need to connect to a network, which may involve wpa_supplicant configuration.

Before doing anything else, I usually pull down the latest sources and rebuild world and the kernel first.  This ensures my system is up-to-date.  There are plenty of guides on doing this, so I won’t waste the space describing how to do it here.

You’ll also need to obtain the ports collection.  Again, there are plenty of guides on doing this.

Installing GRUB

The grub-efi port will install a version of GRUB capable of booting an EFI system.  This port is much easier to use in my opinion than the Gentoo equivalent.  The port is installed as follows:

cd /usr/ports/sysutils/grub2-efi
make install clean

At this point, you’ll need to create a grub.cfg file.  The grub-mkconfig command gets a good start, but you’ll inevitably need to edit it.  You can also just use the following file directly (make sure it’s at /boot/grub/grub.cfg):

insmod part_gpt
insmod zfs

menuentry 'FreeBSD' --class freebsd --class bsd --class os {
  search.fs_label data ZFS_PART
  echo "Loading FreeBSD Kernel..."
  kfreebsd ($ZFS_PART)/freebsd/@/boot/kernel/kernel
  kfreebsd_loadenv ($ZFS_PART)/freebsd/@/boot/device.hints
  kfreebsd_module_elf ($ZFS_PART)/freebsd/@/boot/kernel/opensolaris.ko
  kfreebsd_module_elf ($ZFS_PART)/freebsd/@/boot/kernel/acl_nfs4.ko
  kfreebsd_module_elf ($ZFS_PART)/freebsd/@/boot/kernel/zfs.ko
  set kFreeBSD.vfs.root.mountfrom=zfs:data/freebsd
  set kFreeBSD.vfs.root.mountfrom.options=rw

menuentry 'Gentoo Linux' {
  search.fs_label data ZFS_PART
  linux ($ZFS_PART)/gentoo@/boot/kernel dozfs=force root=ZFS=data/gentoo
  initrd ($ZFS_PART)/gentoo@/boot/initramfs

Note that we’ve created an entry for Gentoo, though it doesn’t yet exist.  Last, you’ll need to mount your EFI system partition and install GRUB:

mount -t msdosfs /dev/ada0p1 /mnt
grub-install --efi-directory=/mnt --disk-module=efi

This will install the GRUB boot program to /efi/grub/grub.efi on the EFI system partition.  You’ll need to copy it into place.  However, I recommend making a backup of your FreeBSD native bootloader first!

cp /mnt/efi/BOOT/BOOTX64.EFI /mnt/efi/BOOT/BOOTX64.BAK

This will simplify the recovery for you if things go wrong.  Now, copy the GRUB boot loader into place:

cp /mnt/efi/grub/grub.efi /mnt/efi/BOOT/BOOTX64.EFI

You should test your GRUB bootloader once to make sure it works by rebooting the system and booting into FreeBSD.  Don’t try to boot into Gentoo yet, as nothing is there!

Gentoo Installer

Your next task is to install the Gentoo base system.  Gentoo installation is done manually via the command line.  A guide is provided by the Gentoo Handbook.  Note that because you’re using ZFS as a root filesystem, you’ll need to do things a bit differently, and you will have to use genkernel to install your kernel!

Mount the Filesystems

As with FreeBSD, you’ll need to mount the filesystems:

zpool import -f data
mount -t zfs data/gentoo /mnt/gentoo
mkdir /mnt/gentoo/tmp
mkdir /mnt/gentoo/usr
mkdir /mnt/gentoo/var
mount -t zfs data/gentoo/tmp /mnt/gentoo/tmp
mount -t zfs data/gentoo/usr /mnt/gentoo/usr
mount -t zfs data/gentoo/var /mnt/gentoo/var

Now follow the Gentoo install steps and everything should go smoothly.

Creating the fstab

As with the FreeBSD system, you’ll need to create an /etc/fstab file.  The file looks similar to the FreeBSD version, but with the gentoo filesystems mounted relative to root and the FreeBSD filesystems mounted relative to /freebsd:

data/freebsd/tmp /freebsd/tmp zfs rw 0 0
data/freebsd/usr /freebsd/usr zfs rw 0 0
data/freebsd/usr/include /freebsd/usr/include zfs rw 0 0
data/gentoo/tmp /tmp zfs rw 0 0
data/gentoo/usr /usr zfs rw 0 0

Again, I’ve left out the repetitive portions of the file.

Building the Kernel, ZFS Modules, and initramfs

As we are booting from a root ZFS filesystem, you’ll need to set up a kernel with ZFS support.  You can find a guide to doing this here (skip down to the “Configuring the Kernel” section and go from there).

Note that I’ve set up the GRUB installation to go after /boot/kernel and /boot/initramfs.  Gentoo by default installs its kernel to /boot/kernel-<version information>, and the same with initramfs.  You’ll need to create symlinks with the name /boot/kernel and /boot/initramfs, or else copy the files into the right place yourself.

Final Gentoo Installation

Before you reboot, you’ll need to make sure that you’re read.  Here is a checklist of things I forgot and had to reboot into the installer to do:

  • Set a root password so you can actually log in
  • Install the ports to use wireless
  • Don’t miss a volume in /etc/fstab (if you miss /var, portage will get very confused)

Boot into Gentoo

You should now be able to boot into Gentoo directly from GRUB.  Congratulations!  You now have a dual-boot, single-ZFS system!  The last thing you’ll want to do before you create an user home directories is create a ZFS dataset for /home.  In the Gentoo system, do the following:

rm /home || rmdir /home
rm /freebsd/home || rmdir /freebsd/home
mkdir /home
mkdir /freebsd/home
zfs create -o mountpoint=/home -o exec=on -o setuid=off -o compression=lz4 data/home

You may also want to create datasets for specific users’ home directories (or even subdirectories thereof).  Note that we’ve set the mountpoint to /home.  This will cause the ZFS mountpoint functionality to mount those datasets, so there’s no need to add an fstab entry.


I have found this setup to be quite powerful and flexible, especially for kernel hackers and driver developers.  The following are some of the benefits of the setup:

  • Ready access to a full Linux system, including kernel sources from the FreeBSD system.
  • Convenient switching between systems for experimentation purposes
  • Effective recovery mechanism if one system gets broken

There’s also some interesting room for work with the FreeBSD Linux emulation layer here.  Normally, the FreeBSD Linux emulation ports install a minimal set of Linux packages.  I don’t know the subsystem well enough to do it myself, but I imagine there’s serious potential in having a full Linux installation at your disposal.

Design Sketch for a Quantum-Safe Encrypted Filesystem

Almost all disk encryption systems today follow a similar design pattern.  Symmetric-key block ciphers are used, with the initialization vector being derived entirely from the block index to which the data is stored.  Often times, the disk is broken up into sections, each of which has its own key.  However, the point of this is that the key and IV are static across any number of writes.

This preserves the atomicity of writes and allows the design to work at the block layer as opposed to the filesystem layer.  However, it also restricts the modes of operation to those that are strong against reuse of IVs.  Typically, this means CBC mode.  This block-level design also makes it quite difficult to integrate a MAC.  Modes like AES-XTS go some distance to mitigating this, and the problem can be mitigated completely by using a filesystem with inherent corruption-resistance like ZFS.

The problem is that this scheme completely prohibits the use of stream ciphers such as ChaCha20 or modes of operation such as CTR or OFB that produce stream cipher-like behavior.  This would be a footnote but for recent results that demonstrate a quantum period-finding attack capable of breaking basically all modes other than CTR or OFB.  This suggests that to implement quantum-safe encrypted storage, we need to come up with a scheme capable of using stream ciphers.

The Problem of Disk Encryption

The fundamental problem with using stream ciphers for block-layer disk encryption stems from the fact that the initialization vector (and ideally the key) must be changed every time the block is written, and this key must be available at an arbitrarily later time in order to read.

In general, there are basically three ways to manage keys in the context of disk encryption:

  1. Derive the key and IV from the block index
  2. Store the keys in a separate location on disk, look this up when needed
  3. Alter the interface for block IO to take a key as a parameter

Most current disk encryption schemes use option 1; however, this ends up reusing IVs, which prohibits the use of stream ciphers (and modes like CTR and OFB).  Option 2 guarantees that we have a unique IV (and key, if we want it) every time we write to a given block; we simply change keys and record this in our key storage.  The price we pay for this is atomicity: every incoming block write requires a write to two separate disk blocks.  This effectively undermines even atomic filesystems like ZFS.  The only example of this sort of scheme of which I am aware is FreeBSD’s older GBDE system.

Option 3 eschews the problem to someone else.  Of course, this means they have to solve the problem somehow.  The only way this ends up not being wholly equivalent to options 1 or 2 is if the consumer of the block-layer interface (the filesystem) somehow organizes itself in a way that a key and IV are always readily available whenever a read or write is about to take place.  This of course, requires addressing full-disk encryption in the filesystem itself.  It also places certain demands on the design of the filesystem.

Atomic Snapshot Filesystems

Atomic filesystems are designed in such a way that all I/O operations appear to be atomic.  With regard to writes, this means that the sort of filesystem corruption that necessitates tools like fsck cannot happen.  Either an operation takes place, or it does not.

Of course, a given write operation may actually perform many block writes; however, the filesystem’s on-disk data structures are carefully designed in such a way that one single write causes all of the operations that lead up to it to “take effect” at once.  Typically, this involves building up a number of “shadow” objects representing the new state, then switching over to them in a single write.

Note that in this approach, we effectively get snapshots for free.  We have a data structure consisting of a mutable spine that points to a complex but immutable set of data structures.  We never overwrite anything until the single operation that updates the mutable spine, causing the operation to take effect.

Atomic Key/IV Updates

The atomic snapshot filesystem design provides a way to effectively change the keys and IVs for every node of a filesystem data structure every time it is written.  Because we are creating a shadow data structure, then installing it with a single write, it is quite simple to generate new keys or IVs every time we create a node in this shadow structure.  Conversely, because the filesystem is atomic, and every node contains the keys and IVs for any node to which it points, anyone traversing the filesystem always has the information they need to decrypt any object they can reach.

This scheme has advantages over conventional disk or filesystem encryption.  Unlike conventional disk encryption, each filesystem object has its own key and IV, and these are uniquely generated every time a write takes place.  Nothing about the key and IV can be inferred by any attacker looking at an arbitrary disk block.  Unlike conventional filesystem encryption which typically only encrypts file contents, everything is encrypted.

Possible ZFS Extension

The ZFS filesystem is a highly advanced filesystem and volume management scheme that provides fully atomic operations and snapshots.  I am admittedly not familiar enough with its workings to know for absolute certain whether the scheme I describe above could be added to it, but I am fairly confident that it could.  I am also aware that ZFS provides an encryption system already, but I am also fairly confident that it is not equivalent to the scheme I describe above.

ZFS would also need to be extended to support a broader range of ciphers and modes of operation to take advantage of this scheme.  Support for CTR and OFB modes are absolutely essential, of course.  I would also recommend support for ciphers beyond AES.  Camellia and ChaCha20 would make good additions, among others.


Quantum-safe disk encryption is arguably not as critical to develop as quantum-safe encryption for network communications.  With network communications, it is reasonable to assume that all traffic is being recorded and will be subject to quantum attacks once those become available.  The same it not true of disk storage.  However, the technology does need to be developed, and the recent results about the period-finding attack on symmetric cipher modes demonstrate a workable attack against nearly all disk encryption schemes.

I would urge all filesystem projects to consider the scheme I’ve laid out and integrate concerns for quantum-safe encryption into their design.

As a final note, should anyone from Illumos run across this blog, I’d be more than willing to discuss more details of this scheme with them.

ZFS Support for EFI Now in FreeBSD

Sometime last year, I started working on a patch to add ZFS support to the UEFI boot facilities for FreeBSD.

Backstory: I’ve been a FreeBSD fan and user since my junior year of undergrad (2003), and I run it as my OS whenever I can.  I first started looking into UEFI support as a GSoC project.  Unfortunately, I had to drop the project due to a combination of a sudden job search and my grandfather’s brain cancer diagnosis.

Fast forward a few years, and I circled back to see what remained to be done on the UEFI front.  The boot process was there, but only for UFS.  So over the holidays, I started poking around to see what could be done.

I started out by refactoring boot1 (the program that resides in the EFI partition and pulls loader off the main filesystem and runs it), putting it into a more modular form to support multiple filesystem modules.  I then started writing the ZFS module.  I hit a tipping point some time in April, and got it working completely shortly thereafter.

The next task was loader itself.  This proved trickier, but I eventually figured out what needed to be done.  To my delight, the modified loader worked fine with the GRUB2 bootloader as well as FreeBSD’s boot1.

For most of the rest of the year, it’s been passed around and used by various people and was picked up by NextBSD and PCBSD.  It entered the formal review process in late autumn, and several people contributed changes that helped out immensely in the integration effort.  In particular, several people addressed stylistic issues (I am not terribly familiar with FreeBSD’s style guide) and integrated libstand support (which I had thought to be a problem due to the need for Win32 ABI binaries in EFI).

I was informed on the way home from the gym that it’s been committed to HEAD, and will hopefully make it into 10.3.  I’m glad to see it now officially in FreeBSD, and I’m grateful to the people who helped out with the integration.

I have future plans in this arena, too.  I deliberately modularized the boot1 program in preparation for some other efforts.  First, I plan to look into adding GELI (the full-disk encryption mechanism for FreeBSD) support.  I would also like to see support for checking cryptographic signatures of loader and kernel at boot-time (I’ve heard others are working on something like that).  In the very long run, I’d like to see a completely trusted custody chain from boot to kernel, but that is something that will take multiple steps to realize.