Slides from my BSDCan Talk

These are the slides from by BSDCan talk on a system-level public-key trust system for FreeBSD.

Slides from vBSDcon

These are the slides from my vBSDcon talk on GELI work.

Design of a Trust System for FreeBSD

About a month ago, I started a discussion on freebsd-hackers and freebsd-security about a system for signed executables, with a focus on signed kernels and kernel modules.  This is part of a larger agenda of mine to equip FreeBSD with OS-level tamper resistance features.

While the initial use of this is for signing the kernel and its modules, and checking signatures during the loader process as well as at runtime when kernel modules are loaded.  However, it is desirable to build a system that is capable of growing in likely directions, such as executable and library signing.

This article details the current state of the design of this system.

Desiderata

I originally outlined a number of goals for this system:

1. Be able to check for a correct cryptographic signature for any kernel or modules loaded at boot time for some platforms (EFI at a minimum)
2. Be able to check for a correct cryptographic signature for any kernel module loaded during normal operations (whether or not to do this could be controlled by a sysctl, securelevel, or some similar mechanism)
3. Work with what’s in the base system already and minimize new additions (ideally, just a small utility to sign executables)
4. Minimize administrative overhead and ideally, require no changes at all to maintain signed kernel/modules
5. Have a clear path for supporting signed executables/libraries.
6. The design must support the case where a system builds locally and uses its own key(s) for signing kernels and modules (and anything else) and must allow the administrator complete control over which key(s) are valid for a given system (ie. no “master keys” controlled by central organizations)
7. The design must allow for the adoption of new ciphers (there is an inevitable shift to post-quantum ciphers coming in the near future)

I also specified a number of non-goals:

• Hardware/firmware-based attacks are considered out-of-scope (there is no viable method for defending against them at the OS level)
• Boot platforms that don’t provide their own signature-checking framework up to loader/kernel can’t be properly secured, and are considered out-of-scope
• Boot platforms that impose size restrictions prohibiting incorporation of RSA and ED25519 crypto code (ex. i386 BIOS) are considered out-of-scope
• GRUB support is desirable, however it is not necessary to support GRUB out-of-the-box (meaning a design requiring reasonable modifications to GRUB is acceptable

Considerations

There are several considerations that should weigh in on the design.

FreeBSD Base System

Unlike linux, FreeBSD has a base operating system which contains a number of tools and libraries which provide a set of operating system utilities.  Most notably, the base system contains the OpenSSL (or in some cases, LibreSSL) crypto suite.  This includes an encryption library as well as tools capable of creating and managing key-pairs and other cryptographic data in a variety of formats.

Additionally, the FreeBSD base system contains libelf, which is a library that provides mechanisms for manipulating ELF binaries.  Additionally, the base system provides the binutils suite, including objcopy, which are command-line tools capable of manipulating ELF binaries.

Note that only some of these components (namely the signelf tool) exist at the present; the rest of the components exist only as man pages that describe them at present.

Public-Key Cryptography

The FreeBSD kernel does not currently incorporate code for public-key cryptography, and direct incorporation of OpenSSL into the kernel has proven infeasible.  Additionally, parsing code needs to be incorporated into the kernel for any formats that are used.  Options here include incorporation of code from the NaCl library, which provides a very lightweight implementation of both RSA 4096 and Ed25519, as well as creating a minimal library out of code harvested from OpenSSL or LibreSSL.

A note on elliptic curve cryptography: the state of support for safe elliptic curves is sad.  In my drafts of the man pages, I have mandated that the only acceptable curves are those that satisfy the security properties described by the SafeCurves project.  At this time, these include M-221, E-222, Curve1174, Curve25519, E-382, M-383, Curve383187, Curve41417, Goldilocks-448, M-511, and E-521.  Unfortunately, none of these is supported by OpenSSL at this time, though Curve25519 support is supposedly coming soon.  However, I would prefer to write specs that mandate the right curves (and thus put pressure on crypto libraries) than cave to using bad ones.

Modifications to GRUB

GRUB provides the best option for FreeBSD coreboot support at this time.  It also provides an existing mechanism for signing binaries.  However, this mechanism is deficient in two ways.  First, it relies on external signatures, which would complicate administration and require modification of virtually all installer programs, as well as run the risk of stale signatures.  Second, it relies on the gnupg toolset, which is not part of the FreeBSD base system.  Thus, it is inevitable that GRUB will need to be patched to support the signed executables proposed by this design.  However, we should make efforts to keep the necessary changes as minimal as possible.

Signing and Trust System Design

The signing and trust system consists of a number of components, some of which are standards, some of which are interfaces, and some of which are tools.  The core feature, of course, is the signed ELF convention.  The signelf tool provides a one-stop tool for signing large numbers of executables.  The trust system provides a system-level mechanism for registering and maintaining verification keys that are used to check signatures on kernel modules.  Finally, the portable verification library provides a self-contained code package that can be dropped into the kernel, the loader, or a third-party codebase like GRUB.

Note that this design is not yet implemented, so it may be subject to change.  Also, it has not yet undergone review on the FreeBSD lists, so it should be considered more of a proposal.

Signed ELF Binaries

The ELF format is very flexible, and provides a generic mechanism for storing metadata.  The signed ELF convention utilizes this to store signatures in a special section within the binary itself.  A signed ELF binary contains a section named .sign, which contains a detached PKCS#7 signature in DER encoding for the file.  This signature is computed (and checked) on the entire file, with the .sign section itself being replaced by zero data of equal size and position.

Signing an ELF binary is somewhat involved, as it requires determining the size of a signature, creating a new section (along with its name), recomputing the ELF layout, computing the signature, and writing it into the section.  Checking a signature is considerably simpler: it involves merely copying the signature, overwriting the .sign section with zeros, and then checking the signature against the  entire file.

The PKCS#7 format was chosen because it is an established standard which supports detached signatures as well as many other kinds of data.  The signatures generated for signed ELF files are minimal and do not contain certificates, attributes, or other data (a signature for RSA-4096 is under 800 bytes); however, the format is extensible enough to embed other data, allowing for future extensions.

The signelf Tool

Signed ELF binaries can be created and checked by adroit usage of the objcopy and openssl command-line tools.  This is quite tedious, however.  Moreover, there are certain use cases that are desirable, like signing a batch of executables using an ephemeral key, discarding the key, and generating a certificate for verification.  The signelf tool is designed to be a simplified mechanism for signing batches of executables which provides this additional functionality.  It is a fairly straightforward use of libelf and OpenSSL, and should be able to handle the binaries produced by normal compilation.  Additionally, the signelf tool can verify signed ELF files.  The signelf code is currently complete, and works on a kernel as well as modules.

The Trust System

In order to check signatures on kernel modules (and anything else), it is necessary to establish and maintain a set of trusted verification keys in the kernel (as well as in the boot loader).  In order for this system to be truly secure, at least one trust root key must be built into the kernel and/or the boot loader, which can then be used to verify other keys.  The trust system refers to the combination of kernel interfaces, standard file locations, and conventions that manage this.

System Trust Keys and Signing Keys

The (public) verification keys used to check signatures as well as the (private) signing keys used to generate signatures are kept in the /etc/trust/ directory.  Verification keys are stored in /etc/trust/certs, in the X509 certificate format, and private keys are stored in /etc/trust/keys in the private key format.  Both are stored in the PEM encoding (as is standard with many OpenSSL applications).

There is no requirement as to the number, identity, or composition of verification or signing keys.  Specifically, there is not and will never be any kind of mandate for any kind of verification key not controlled by the owner of the machine.  The trust system is designed to be flexible enough to accommodate a wide variety of uses, from machines that only trust executables built locally, to ones that trust executables built on an in-house machine only, to those that trust executables built by a third party (such as the FreeBSD foundation), or any combination thereof.

The preferred convention, however, is to maintain a single, per-machine keypair which is then used to sign any additional verification keys.  This keypair should be generated locally for each machine, and never exported from the machine.

Trust Keys Library

Keys under /etc/trust/certs will be converted into C code constants and subsequently compiled into a static library providing the raw binary data for the keys during the buildworld process.  This provides the mechanism for building keys into the kernel, loader, and other components.  These keys are known as trust root keys, as they provide the root set for all trusted keys.

Kernel Trust Interface

The kernel trust interface provides access to the set of verification keys trusted by the kernel.  This consists of an in-kernel interface as well as a user-facing device interface.  The in-kernel interface looks like an ordinary key management system (KMS) interface.  The device interface provides two primary mechanisms: access to the current set of trusted keys and the ability to register new keys or revoke existing ones.

Access to the existing database is accomplished through a read-only device node which simply outputs all of the existing trusted keys in PEM-encoded X509 format.  This formatting allows many OpenSSL applications to use the device node itself as a CA root file.  Updating the key database is accomplished by writing to a second device node.  Writing an X509 certificate signed by one of the existing trusted keys to this device node will cause the key contained in the certificate to be added to the trusted key set.  Writing a certificate revocation list (CRL) signed by a trusted key to the device node will revoke the keys in the revocation list as well as any keys whose signature chains depend on them.  Trust root keys cannot be revoked, however.

This maintains the trusted key set in a state where any trusted key has a signature chain back to a trust root key.

Portable Verification Library

The final piece of the system is the portable verification library.  This library should resemble a minimal OpenSSL-like API that performs parsing/encoding of the necessary formats (PKCS#7, X509, CRL), or a reduced subset thereof and public-key signature verification.  I have not yet decided whether to create this from harvesting code from OpenSSL/LibreSSL or write it from scratch (with code from NaCl), but I’m leaning toward harvesting code from LibreSSL.

Operation

The trust system performs two significant roles in the system as planned, and can be expanded to do more things in the future.  First, it ensures that loader only loads kernels and modules that are signed.  Second, it can serve as a kind of system-wide keyring (hence the device node that looks like a typical PEM-encoded CAroot file for OpenSSL applications).  The following is an overview of how it would operate in practice.

Signature Checking in the loader

In an EFI environment, boot1.efi and loader.efi have a chain of custody provided by the EFI secure boot framework.  This is maintained from boot1.efi to loader.efi, because of the use of the EFI loaded image interface.  The continuation of the chain of custody must be enforced directly by loader.efi.  To accomplish this, loader will link against the trust key library at build time to establish root keys.  These in turn can either be used to check the kernel and modules directly, or they can be used to check a per-kernel key (the second method is recommended; see below).

Per-Kernel Ephemeral Keys

The signelf utility was designed with the typical kernel build process in mind.  The kernel and all of its modules reside in a single directory; it’s a simple enough thing to run signelf on all of them as the final build step.  Additionally, signelf can generate an ephemeral key for signing and write out the verification certificate after it finishes.

This gives rise to a use pattern where every kernel is signed with an ephemeral key, and a verification certificate is written into the kernel directory.  This certificate is in turn signed by the local trust root key (signelf does this as part of the ephemeral key procedure).  In this case, the loader first attempts to load the verification certificate for a kernel, then it loads the kernel and all modules.

Signed Configuration Files

The FreeBSD loader relies on several files such as loader.4th, loader.conf, loader.menu, and others that control its behavior in significant ways.  Additionally, one can foresee applications of this system that rely on non-ELF configuration files.  For loader, the simplest solution is to store these files as non-detached PKCS#7 messages (meaning, the message and file contents are stored together).  Thus, loader would look for loader.conf.pk7, loader.4th.pk7, and so on.  A loader built for secure boot would look specifically for the .pk7 files, and would require signature verification in order to load them.

The keybuf Interface

The kernel keybuf interface was added in a patch I contributed in late March 2017.  It is used by GELI boot support to pass keys from the boot phases to the kernel.  However, it was designed to support up to 64 distinct 4096-bit keys without modification; thus it can be used with RSA-4096.  An alternative to linking the trust key library directly into the kernel is to have it receive the trusted root key as a keybuf entry.

This approach has advantages and disadvantages.  The advantage is it allows a generic kernel to be deployed to a large number of machines without rebuilding for each machine.  Specifically, this would allow the FreeBSD foundation to publish a kernel which can make use of local trust root keys.  The primary disadvantage is that the trust root keys are not part of the kernel and thus not guaranteed by the signature checking.  The likely solution will be to support both possibilities as build options.

Key Management

The preferred scheme for trust root keys is to have a local keypair generated on each machine, with the local verification certificate serving as the sole trust root key.  Any vendor keys that might be used would be signed by this keypair and loaded as intermediate keys.  Every kernel build would produce an ephemeral key which would be signed by the local keypair.  Kernel builds originating from an organization would also be signed by an ephemeral key, whose certificate is signed by the organization’s keypair.  For example, the FreeBSD foundation might maintain a signing key, which it uses to sign the ephemeral keys of all kernel builds it publishes.  An internal IT organization might do the same.

It would be up to the owner of a machine whether or not to trust the vendor keys originating from a given organization.  If the keys are trusted, then they are signed by the local keypair.  However, it is always an option to forego all vendor keys and only trust locally-built kernels.

An alternate use might be to have no local signing key, and only use an organizational trust root key.  This pattern is suitable for large IT organizations that produce lots of identical machines off of a standard image.

Conclusion

This design for the trust system and kernel/module signing is a comprehensive system-wide public-key trust management system for FreeBSD.  Its initial purpose is managing a set of keys that are used to verify kernels and kernel modules.  However, the system is designed to address the issues associated with trusted key management in a comprehensive and thorough way, and to leave the door open to many possible uses in the future.

Cohabiting FreeBSD and Gentoo Linux on a Common ZFS Volume

My Librem 15 arrived a while back.  I normally prefer FreeBSD for just about everything, but I need access to a Linux OS running on the Librem platform in order to help me port over some remaining device drivers (namely the BYD mouse and screen brightness).

In order to facilitate this, I’ve installed a setup I developed a while back: that of a dual-booted FreeBSD and Gentoo Linux system living on the same ZFS volume.  This article details how to achieve this setup.

Note that this article is based on the EFI bootloader.  If you insist on legacy BIOS boots, you’ll need to adapt the procedure.

Overview of the Scheme

The scheme is based on a somewhat atypical use of the ZFS filesystem (namely foregoing the mountpoint functionality for the OS datasets in favor of an fstab-based approach) combined with GRUB to achieve a dual-bootable OS

ZFS Overview

The ZFS system differs slightly from the “typical” ZFS setup on both FreeBSD and Linux.  Some datasets (namely the home directories) are shared between both operating systems, but the OS datasets differ in their mount-points depending on which OS we are using, and thus the ZFS-specific mountpoint functionality can’t be effectively used.

In this article, assume that the volume’s name is “data”.

The overall scheme looks something like this:

• data/home is mounted to /home, with all of its child datasets using the ZFS mountpoint system
• data/freebsd and its child datasets house the FreeBSD system, and all have their mountpoints set to legacy
• data/gentoo and its child datasets house the Gentoo system, and have their mountpoints set to legacy as well

OS and GRUB Overview

Both OSes must utilize the /etc/fstab method for mounting most of their filesystems, since we cannot make use of the ZFS mountpoint functionality.  This requires a different fstab for each OS.  Note that the data/home dataset (as well as any other similar dataset) will be mounted using the ZFS mountpoint method, not fstab.

Additionally, both OSes have access to the other OS’ data through a special top-level directory (/freebsd on Gentoo, /gentoo on FreeBSD).

The GRUB bootloader can be used to provide a workable boot selection facility without any serious modification or configuration (other than knowing the magic words to type into the grub.cfg file!)

Setup Procedure

The setup procedure consists of the following steps:

1. Use the FreeBSD installer to create the GPT and ZFS pool
2. Install and configure FreeBSD, with the native FreeBSD boot loader
3. Boot into FreeBSD, create the Gentoo Linux datasets, install GRUB
4. Boot into the Gentoo Linux installer, install Gentoo
5. Boot into Gentoo, finish any configuration tasks

Note that nothing stops you from reversing the procedure, installing Gentoo first, and using its tools.  I just find that GPT creation and GRUB installation go a lot more smoothly on FreeBSD.

Getting Ready

To perform the setup procedure, you’ll need installer memstick images for both OSes.  FreeBSD’s installer can be gotten here; Gentoo’s can be gotten here (use the livedvd ISO).  You’ll also need some means of internet access (of course).

Note that in the case of the Librem 15 or similar laptops which have no ethernet connector, you may need to adopt a slightly modified procedure of installing Gentoo’s wireless tools and wpa_supplicant during the main installation.

FreeBSD Installer

Boot into the FreeBSD installer, go through the boot menu, and select manual partitioning mode.  This will drop you into a shell and ask you to create your partitions and mount everything at /mnt.

Create Partitions and ZFS Pool

The first thing to do is use the gpart tool to create your partitions.  FreeBSD’s man pages are rather good, so you can use “man gpart” to get the guide to the tool.  My procedure on the Librem 15 looks like this:

gpart create -s gpt ada0
gpart create -s gpt ada1
gpart add -t efi -l efi-system -s 200M ada0
gpart add -t freebsd-zfs -l zfs-data ada0
gpart add -t linux-swap -l swap -s 96G ada0
gpart add -t freebsd-zfs -l zfs-data-log -s 16G ada0
gpart add -t freebsd-zfs -l zfs-data-cache ada0

Then create a ZFS pool with the new partitions, and format the EFI system partition with the DOS filesystem (seriously, why do we still use that thing?):

newfs_msdos /dev/ada0p1
zpool create -m legacy -o atime=off -o checksum=sha256 data /dev/ada0p2 log /dev/ada0p2 cache /dev/ada0p3

Note that we’ve turned off atime (which reduces disk write traffic considerably) and set the checksum algorithm to sha256.

The ada1 disk is a SSD I had installed.  If you don’t have an SSD, it doesn’t make any sense to have a log or a cache.  The 16GB intent log is way overkill, but it reduces the strain on the device.  Note that we set the root dataset’s mountpoint to “legacy”.

Note that linux has its own swap format, so we can’t share the swap device.

Create the ZFS Datasets

Now that you have a ZFS pool, the first thing you’ll need to do is create the datasets.   Start by creating the FreeBSD root and mounting it (note that it will inherit the “legacy” mountpoint from its parent):

zfs create -o compression=lz4 data/freebsd
mount -t zfs data/freebsd /mnt/

We need to create some mountpoint directories:

mkdir /mnt/home
mkdir /mnt/gentoo/
mkdir /mnt/tmp
mkdir /mnt/usr
mkdir /mnt/var

I use a fairly elaborate ZFS scheme, which applies different executable, setuid, and compression properties for certain directories.  This achieves a significant compression ratio, effectively increasing the size of my disks:

zfs create -o exec=on -o setuid=off -o compression=off data/freebsd/tmp
zfs create -o exec=on -o setuid=on -o compression=lz4 data/freebsd/usr
zfs create -o exec=off -o setuid=off -o compression=gzip data/freebsd/usr/include
zfs create -o exec=on -o setuid=off -o compression=lz4 data/freebsd/usr/lib
zfs create -o exec=on -o setuid=off -o compression=lz4 data/freebsd/usr/lib32
zfs create -o exec=on -o setuid=off -o compression=gzip data/freebsd/usr/libdata
zfs create -o exec=on -o setuid=on -o compression=lz4 data/freebsd/usr/local
zfs create -o exec=on -o setuid=off -o compression=gzip data/freebsd/usr/local/etc
zfs create -o exec=off -o setuid=off -o compression=gzip data/freebsd/usr/local/include
zfs create -o exec=on -o setuid=off -o compression=lz4 data/freebsd/usr/local/lib
zfs create -o exec=on -o setuid=off -o compression=lz4 data/freebsd/usr/local/lib32
zfs create -o exec=on -o setuid=off -o compression=gzip data=freebsd/usr/local/libdata
zfs create -o exec=on -o setuid=off -o compression=gzip data/freebsd/usr/local/share
zfs create -o exec=off -o setuid=off -o compression=off data/freebsd/usr/local/share/info
zfs create -o exec=off -o setuid=off -o compression=off data/freebsd/usr/local/share/man
zfs create -o exec=on setuid=on -o compression=lz4 data/freebsd/obj
zfs create -o exec=on -o setuid=on -o compression=lz4 data/freebsd/usr/ports
zfs create -o exec=off -o setuid=off -o compression=lz4 data/freebsd/usr/ports
zfs create -o exec=on -o setuid=off -o compression=gzip data/freebsd/usr/share
zfs create -o exec=off -o setuid=off -o compression=off data/freebsd/usr/share/info
zfs create -o exec=off -o setuid=off -o compression=off data/freebsd/usr/share/man
zfs create -o exec=off -o setuid=off -o compression=gzip data/freebsd/usr/src
zfs create -o exec=off -o setuid=off -o compression=lz4 data/freebsd/var
zfs create -o exec=off -o setuid=off -o compression=off data/freebsd/var/db
zfs create -o exec=off -o setuid=off -o compression=lz4 data/freebsd/var/db/pkg
zfs create -o exec=off -o setuid=off -o compression=gzip data/freebsd/var/log
zfs create -o exec=off -o setuid=off -o compression=off data/freebsd/var/empty
zfs create -o exec=off -o setuid=off -o compression=gzip data/freebsd/var/mail
zfs create -o exec=on -o setuid=off -o compression=off data/freebsd/var/tmp

Because FreeBSD is pretty strict about where certain files go, this scheme works pretty well.  You could of course continue to subdivide it up to your heart’s desire, creating more subdirectories in /usr/share and such.

For Gentoo, you’ll probably want a simpler scheme, as Linux tends to be much sloppier about the locations of things:

zfs create -o exec=on -o setuid=off -o compression=off data/gentoo/tmp
zfs create -o exec=on -o setuid=on -o compression=lz4 data/gentoo/usr
zfs create -o exec=off -o setuid=off -o compression=lz4 data/gentoo/var

A Gentoo master might be able to subdivide this up as I’ve done with FreeBSD.

The final task is to mount all these filesystems manually with the following command template:

mount -t zfs data/freebsd/<path> /mnt/<path>

This is necessary, as all the mount-points will be “legacy”.  I won’t waste space by showing all the commands here.

Install and Configure the FreeBSD system

Now type “exit”, which will drop you back into the FreeBSD installer, with everything mounted at /mnt/.  The remainder of the installation procedure is straightforward.  However, you’ll need to opt to go to a shell for two final configuration tasks.

Go to the shell, then enter into the new FreeBSD system

chroot /mnt

Create the fstab

Since we mount the vast majority of the ZFS datasets to different paths in each OS, we’ll need to create an /etc/fstab file for them.  The following fstab will mount all the datasets to the right locations:

data/freebsd/tmp /tmp zfs rw 0 0
data/freebsd/usr /usr zfs rw 0 0
data/freebsd/usr/include /usr/include zfs rw 0 0
...
data/gentoo/ /gentoo zfs rw 0 0
data/gentoo/tmp /gentoo/tmp zfs rw 0 0
...
proc /proc procfs rw 0 0

Note that I’ve left out all a number of the entries.  You’ll have to map each dataset to its proper path as shown above.

Install the FreeBSD Native Bootloader

We’ll need the FreeBSD bootloader to get into the system for the first time.  Install it to the with the following procedure:

mount -t msdosfs /dev/ada0p1 /mnt
mkdir /mnt/efi
mkdir /mnt/efi/BOOT
cp /boot/boot1.efi /mnt/efi/BOOT/BOOTX64.EFI

The last thing you’ll need to do is set the bootfs parameter on the zpool, so the FreeBSD bootloader will pick the right dataset:

zpool set -o bootfs=data/freebsd data

You may also need to set the bootme flag on the EFI system partition for some hardware:

gpart set -a bootme -i 1 ada0

Your system is now ready to boot into the OS directly.

FreeBSD Main OS

You should now be able to boot into FreeBSD directly.  You’ll need to connect to a network, which may involve wpa_supplicant configuration.

Before doing anything else, I usually pull down the latest sources and rebuild world and the kernel first.  This ensures my system is up-to-date.  There are plenty of guides on doing this, so I won’t waste the space describing how to do it here.

You’ll also need to obtain the ports collection.  Again, there are plenty of guides on doing this.

Installing GRUB

The grub-efi port will install a version of GRUB capable of booting an EFI system.  This port is much easier to use in my opinion than the Gentoo equivalent.  The port is installed as follows:

cd /usr/ports/sysutils/grub2-efi
make install clean

At this point, you’ll need to create a grub.cfg file.  The grub-mkconfig command gets a good start, but you’ll inevitably need to edit it.  You can also just use the following file directly (make sure it’s at /boot/grub/grub.cfg):

insmod part_gpt
insmod zfs

menuentry 'FreeBSD' --class freebsd --class bsd --class os {
  search.fs_label data ZFS_PART
  echo "Loading FreeBSD Kernel..."
  kfreebsd ($ZFS_PART)/freebsd/@/boot/kernel/kernel
  kfreebsd_loadenv ($ZFS_PART)/freebsd/@/boot/device.hints
  kfreebsd_module_elf ($ZFS_PART)/freebsd/@/boot/kernel/opensolaris.ko
  kfreebsd_module_elf ($ZFS_PART)/freebsd/@/boot/kernel/acl_nfs4.ko
  kfreebsd_module_elf ($ZFS_PART)/freebsd/@/boot/kernel/zfs.ko
  set kFreeBSD.vfs.root.mountfrom=zfs:data/freebsd
  set kFreeBSD.vfs.root.mountfrom.options=rw
}

menuentry 'Gentoo Linux' {
  search.fs_label data ZFS_PART
  linux ($ZFS_PART)/gentoo@/boot/kernel dozfs=force root=ZFS=data/gentoo
  initrd ($ZFS_PART)/gentoo@/boot/initramfs
}

Note that we’ve created an entry for Gentoo, though it doesn’t yet exist.  Last, you’ll need to mount your EFI system partition and install GRUB:

mount -t msdosfs /dev/ada0p1 /mnt
grub-install --efi-directory=/mnt --disk-module=efi

This will install the GRUB boot program to /efi/grub/grub.efi on the EFI system partition.  You’ll need to copy it into place.  However, I recommend making a backup of your FreeBSD native bootloader first!

cp /mnt/efi/BOOT/BOOTX64.EFI /mnt/efi/BOOT/BOOTX64.BAK

This will simplify the recovery for you if things go wrong.  Now, copy the GRUB boot loader into place:

cp /mnt/efi/grub/grub.efi /mnt/efi/BOOT/BOOTX64.EFI

You should test your GRUB bootloader once to make sure it works by rebooting the system and booting into FreeBSD.  Don’t try to boot into Gentoo yet, as nothing is there!

Gentoo Installer

Your next task is to install the Gentoo base system.  Gentoo installation is done manually via the command line.  A guide is provided by the Gentoo Handbook.  Note that because you’re using ZFS as a root filesystem, you’ll need to do things a bit differently, and you will have to use genkernel to install your kernel!

Mount the Filesystems

As with FreeBSD, you’ll need to mount the filesystems:

zpool import -f data
mount -t zfs data/gentoo /mnt/gentoo
mkdir /mnt/gentoo/tmp
mkdir /mnt/gentoo/usr
mkdir /mnt/gentoo/var
mount -t zfs data/gentoo/tmp /mnt/gentoo/tmp
mount -t zfs data/gentoo/usr /mnt/gentoo/usr
mount -t zfs data/gentoo/var /mnt/gentoo/var

Now follow the Gentoo install steps and everything should go smoothly.

Creating the fstab

As with the FreeBSD system, you’ll need to create an /etc/fstab file.  The file looks similar to the FreeBSD version, but with the gentoo filesystems mounted relative to root and the FreeBSD filesystems mounted relative to /freebsd:

data/freebsd/tmp /freebsd/tmp zfs rw 0 0
data/freebsd/usr /freebsd/usr zfs rw 0 0
data/freebsd/usr/include /freebsd/usr/include zfs rw 0 0
...
data/gentoo/tmp /tmp zfs rw 0 0
data/gentoo/usr /usr zfs rw 0 0
...

Again, I’ve left out the repetitive portions of the file.

Building the Kernel, ZFS Modules, and initramfs

As we are booting from a root ZFS filesystem, you’ll need to set up a kernel with ZFS support.  You can find a guide to doing this here (skip down to the “Configuring the Kernel” section and go from there).

Note that I’ve set up the GRUB installation to go after /boot/kernel and /boot/initramfs.  Gentoo by default installs its kernel to /boot/kernel-<version information>, and the same with initramfs.  You’ll need to create symlinks with the name /boot/kernel and /boot/initramfs, or else copy the files into the right place yourself.

Final Gentoo Installation

Before you reboot, you’ll need to make sure that you’re read.  Here is a checklist of things I forgot and had to reboot into the installer to do:

• Set a root password so you can actually log in
• Install the ports to use wireless
• Don’t miss a volume in /etc/fstab (if you miss /var, portage will get very confused)

Boot into Gentoo

You should now be able to boot into Gentoo directly from GRUB.  Congratulations!  You now have a dual-boot, single-ZFS system!  The last thing you’ll want to do before you create an user home directories is create a ZFS dataset for /home.  In the Gentoo system, do the following:

rm /home || rmdir /home
rm /freebsd/home || rmdir /freebsd/home
mkdir /home
mkdir /freebsd/home
zfs create -o mountpoint=/home -o exec=on -o setuid=off -o compression=lz4 data/home

You may also want to create datasets for specific users’ home directories (or even subdirectories thereof).  Note that we’ve set the mountpoint to /home.  This will cause the ZFS mountpoint functionality to mount those datasets, so there’s no need to add an fstab entry.

Conclusions

I have found this setup to be quite powerful and flexible, especially for kernel hackers and driver developers.  The following are some of the benefits of the setup:

• Ready access to a full Linux system, including kernel sources from the FreeBSD system.
• Convenient switching between systems for experimentation purposes
• Effective recovery mechanism if one system gets broken

There’s also some interesting room for work with the FreeBSD Linux emulation layer here.  Normally, the FreeBSD Linux emulation ports install a minimal set of Linux packages.  I don’t know the subsystem well enough to do it myself, but I imagine there’s serious potential in having a full Linux installation at your disposal.

FreeBSD OS-Level Tamper-Resilience

I’ve posted about my work on EFI GELI support.  This project is actually the first step in a larger series of changes that I’ve been sketching out since April.  The goal of the larger effort is to implement tamper-resilience features at the OS level for FreeBSD.  The full-disk encryption capabilities provided by GELI boot support represent the first step in this process.

OS-Level Tamper-Resilience

Before I talk about the work I’m planning to do, it’s worth discussing the goals and the rationale for them.  One of the keys to effective security is an accurate and effective threat model; another is identifying the scope of the security controls to be put in place.  This kind of thinking is important for this project in particular, where it’s easy to conflate threats stemming from vulnerable or malicious hardware with vulnerabilities at the OS level.

Regarding terminology: “tamper-resistance” means the ability of a device to resist a threat agent who seeks to gain access to the device while it is inactive (in a suspended or powered-off state) in order to exfiltrate data or install malware of some kind.  I specifically use the term “tamper-resilience” to refer to tamper-resistance features confined to the OS layer to acknowledge the fact that these features fundamentally cannot defeat threats based on hardware or firmware.

Threat Model

In our threat model, we have the following assets:

• The operating system kernel, modules, and boot programs.
• Specifically, a boot/resume program to be loaded by hardware, which must be stored as plaintext.
• The userland operating system programs and configuration data.
• The user’s data.

We assume a single threat agent with the following capabilities:

• Access and write to any permanent storage medium (such as a disk) while the device is suspended or powered off.
• Make copies of any volatile memory (such as RAM) while the device is suspended.
• Defeat any sort of physical security or detection mechanisms to do so.

Specifically, the following capabilities are considered out-of-scope (they are to be handled by other mechanisms):

• Accessing the device while powered on and in use.
• Attacks based on hardware or firmware tampering.
• Attacks based on things like bug devices, reading EM radiation (van Eyck phreaking), and the like.
• Attacks based on causing users to install malware while using the device.

Thus, the threat model is based on an attacker gaining access to the device while powered-off or suspended and tampering with it at the OS level and up.

It is important to note that hardware/firmware tampering is a real and legitimate threat, and one deserving of effort.  However, it is a separate and parallel concern that requires its own effort.  Moreover, if the OS level has weaknesses, no amount of hardware or firmware hardening can compensate for it.

Tamper-Resilience Plan

The tamper resilience plan is based around the notion of protecting as much data as possible through authenticated encryption, using cryptographic verification to ensure that any part of the boot/resume process whose program must be stored as plaintext is not tampered with, and ensuring that no other data is accessible as plaintext while suspended or powered off.

The work on this breaks down into roughly three phases, one of which I’ve already finished.

Data Protection and Integrity

All data aside from the boot program to be loaded by the hardware (known in FreeBSD as boot1) can be effectively protected at rest by a combination of ZFS with SHA256 verification and the GELI disk encryption scheme.  Full-disk encryption protects data from theft, and combining it with ZFS’ integrity checks based on a cryptographically-secure hash function prevents an attacker from tampering with the contents (this can actually be done even on encrypted data without an authentication scheme in play).

Secure Boot

There is always at least one program that must remain unprotected by full-disk encryption: the boot entry-point program.  Fortunately, the EFI platform provides a mechanism for ensuring the integrity of the boot program.  EFI secure boot uses public-key crypto to allow the boot program to be signed by a private key and verified by a public key that is provided to the firmware.  If the verification fails, then the firmware informs the user that their boot program has been tampered with and aborts the boot.

In an open-source OS like FreeBSD, this presents an effective protection scheme along with full-disk encryption.  On most desktops and laptops, we build the kernel and boot loaders on the machine itself.  We can simply store a machine-specific signing key on the encrypted partition and use it to sign the boot loader for that machine.  The only way an attacker could forge the signature would be to gain access to the signing key, which is stored on an encrypted partition.  Thus, the attacker would have to already have access to the encrypted volume in order to forge a signature and tamper with the boot program.

To achieve the baseline level of protection, we need to ensure that the plaintext boot program is signed, and that it verifies the signature of a boot stage that is stored on an encrypted volume.  Because of the way the EFI boot process works, it is enough to sign the EFI boot1 and loader programs.  The loader program is typically stored on the boot device itself (which would be encrypted), and loaded by the EFI LOAD_IMAGE_PROTOCOL interface, which performs signature verification.  Thus, it should be possible to achieve baseline protection without having to modify boot1 and loader beyond what I’ve already done.

There is, of course, a case for doing signature verification on the kernel and modules.  One can even imagine signature verification on userland programs.  However, this is out-of-scope for the discussion here.

Secure Suspend/Resume

Suspend/resume represents the most significant tamper weakness at the present.  Suspend/resume in FreeBSD is currently only implemented for the suspend-to-memory sleep state.  This means that an attacker who gains access to the device while suspended effectively has access to the device at runtime.  More specifically, they have all of the following:

• Access to the entire RAM memory state
• Sufficient data to decrypt all mounted filesystems
• Sufficient data to decrypt any encrypted swap partitions
• Possibly the signing key for signing kernels

There really isn’t a way to protect a system that’s suspended to memory.  Even if you were to implement what amounts to suspend-to-disk by unmounting all filesystems and booting the kernel and all programs out to an encrypted disk storage, you still resume by starting execution at a specified memory address.  The attacker can just implant malware in that process if they have the ability to tamper with RAM.

Thus, the only secure way to do suspend/resume is to tackle suspend-to-disk support for FreeBSD.  Of course, it also has to be done securely.  The scheme I have in mind for doing so looks something like this:

• Allow users to specify a secure suspend partition and set a resume password.  This can be done with a standard GELI partition.
• Use the dump functionality to write out the entire kernel state to the suspend partition (because we intend to resume, we can’t do the usual trick of dumping to the swap space, as we need the data that’s stored there)
• Alternatively, since the dump is being done voluntarily, it might be possible to write out to a filesystem (normally, dumps are done in response to a kernel panic, so the filesystem drivers are assumed to be corrupted).
• Have the suspend-to-disk functionality sign the dumped state with a resume key (this can be the signing key for boot1, or it can be another key that’s generated during the build process)
• Make boot1 aware of whatever it needs to know for detecting when resuming from disk and have it request a password, load the encrypted dumped state, and resume.

There are, of course, a lot of issues to be resolved in doing this sort of thing, and I imagine it will take quite some time to implement fully.

Going Beyond

Once these three things are implemented, we’d have a baseline of tamper-resilience in FreeBSD.  Of course, there are ways we could go further.  For one, signed kernels and modules are a good idea.  There has also been talk of a signed executable and libraries framework.

Current Status

My GELI EFI work is complete and waiting for testing before going through the integration process.  There are already some EFI signing utilities in existence.  I’m currently testing too many things to feel comfortable about trying out EFI signing (and I want to have a second laptop around before I do that kind of thing!); however, I plan on getting the baseline signed boot1 and loader scheme working, then trying to alter the build process to support automatically generating signed boot1 and loader programs.

The kernel crypto framework currently lacks public-key crypto support, and it needs some work anyway.  I’ve started working on a design for a new crypto library which I intend to replace the boot_crypto code in my GELI work and eventually the code in the kernel.  I’ve also heard of others working on integrating LibreSSL.  I view this as a precursor to the more advanced work like secure suspend/resume and kernel/module signing.

However, we’re currently in the middle of the 11 release process and there are several major outstanding projects (my GELI work, the i915 graphics work).  In general, I’m reluctant to move forward until those things calm down a bit.

FreeBSD Librem Update

I’ve been working on my FreeBSD setup for Purism’s Librem 13 laptop ever since receiving it back in April.  I’m relatively pleased with the way things have progressed, and most of the critical issues have been addressed.  However, the setup still has a way to go in my opinion before it gets to the point of being the “ideal” setup.

Current State

Three of the four critical issues I identified back in April have been addressed:

• Matt Macy’s i915 graphics patch works well on the Librem 13, and I personally made sure that the suspend/resume support works.  The patch is very stable on the Librem, and I’ve only had one kernel panic the entire time testing it.
• The HDMI output Just Works™ with the i915 driver.  Even better, it works for both X11 and console modes.
• Full support for the Atheros 9462 card has been merged in.  I’ve had some occasional issues, but it works for the most part.
• The vesa weirdness is obviated by i915 support, but it was resolved by using the scfb driver.

While not Librem-specific, I have been working primarily on GELI support for EFI.  This is also part of a larger set of tamper-resilience features I have planned to implement for FreeBSD.

Additional Configuration Notes

I’ve played around with the configuration a bit since April, and discovered a few things in the process:

• The mouse is frozen after suspend.  The way to un-stick it is to restart moused (for some reason, this doesn’t work when done in rc.resume), then move your finger on the pad while pressing the Fn+F1 (mouse lock) keys until it un-sticks.
• PulseAudio works fine, but the default settings end up sending output to the HDMI port even if nothing is plugged in.  Simply change the default sink, and it should work fine.  Sound without PulseAudio also works fine, as I previously reported.

Current Plans

I have plans to do more work on this port, especially with my Librem 15 hopefully arriving soon.  I hope to eventually turn this platform into a fully-supported FreeBSD laptop.

Current To-Do’s

There are still a number of items to be addressed, of course.  Some of these are issues, and some are longer-term work I’ve planned.

Sporadic Boot Hang

The only real “issue” at this point is a boot-hang that manifests sporadically, causing boots to hang for several minutes before having a kernel panic.  Fortunately, the machine tends not to experience this problem after rebooting from such a kernel panic.  Thus, it’s an annoying problem, but not a show-stopper.

After observing this phenomenon many times, I suspect it to be a bug in the either the AML execution engine or the AML itself that causes AML execution to go into an infinite recursion before crashing from a stack overflow.  After this happens, the hardware seems to be in an indeterminate state and causes a kernel panic later on.  Resume also sometimes causes the machine to freeze, and I suspect this to be the root cause of that as well.

This ought to be fairly straightforward to hunt down; the only confounding factor is the fact that it is sporadic, and tends to only happen after the machine has been shut down for a time.

Device Support

All the important devices are supported.  Support is missing for some minor things, though:

• Synaptics support is not present, due to no support for the BYD touchpad beyond the regular PS/2 mouse driver.  I am currently starting an effort to port Purism’s work on the Linux PS/2 driver to FreeBSD.
• I also plan to look into support for things like screen brightness adjustment, either bringing over support from the Linux drivers or working out the configs to make the brightness keys work if the driver support is already there.
• A few of the hotkeys work (the mouse lock and the screen blank), others don’t.  Currently non-functional hotkeys include the volume keys, the brightness keys, the bluetooth and wireless keys, the suspend key, and the video output key.  This probably involves writing an ACPI extras driver similar to acpi_ibm and company.

Longer-Term Plans

My longer-term plans mostly revolve around security, integrity, and tamper-resilience.  Some are related to the hardware platform, while others are general FreeBSD features.

FreeBSD Tamper Resilience

My GELI EFI work was the first step in a series of projects I have planned regarding tamper-resilience features at the OS level for FreeBSD.  I’ve hinted at this, and will write a full post on my plans, but here is a sketch of the process:

1. Full-Disk Encryption for EFI (done): This was addressed by my GELI EFI work
2. Secure Boot and Loading: Provide support in EFI boot1/loader and the kernel for signed images.  Wire it in with EFI Secure Boot architecture.
3. Secure Hibernate: Implement the ability to suspend to disk and resume in a secure fashion.  This is the coup-de-gras, as it means the machine’s data is fully encrypted when powered off or suspended.

I am currently looking at building a new, better crypto library, as the current crypto framework is in need of some cleanup.

CoreBoot and TianoCore

Once my Librem 15 arrives and I have two machines to work with, I plan to replace the proprietary BIOS image with CoreBoot and an open-source EFI implementation (likely TianoCore).  In addition to the obvious benefits in terms of openness and control over my own hardware, this affords certain possibilities like baking extra EFI filesystem drivers into the firmware, allowing me to get rid of the unencrypted FAT-formatted EFI system partition.

It also might be possible to boot straight into FreeBSD and avoid the EFI stuff altogether (this is possible with Linux, and leads to super-fast boots).

Disabling Intel ME

I have a good document on disabling the Intel Management Engine altogether.  This is the one goal that Purism was unable to realize for their Librem laptops (which is fine in my opinion; the push for open hardware is a process, and Purism’s success was a major step forward in that process).

The final stroke in all this is to try and set up a method to disable the ME completely.  While this is obviously platform-specific, it would represent a major victory for the forces of openness.

Conclusions

The Librem platform represents the best chance I’ve ever seen of creating the “perfect” FreeBSD setup.  It’s going to take some work to get to the laptop I’ve always wanted, of course, but stay tuned and hopefully we’ll get there eventually.

Design Sketch for LiCl: A Lightweight Cryptography Library

There has been a lot of work on better cryptography libraries in the wake of a number of OpenSSL bugs.  One of the major steps forward in this realm is NaCl, or the Networking and Cryptography Library.  NaCl aims to address the fact that most older crypto libraries are quite difficult to use, and misuse is often the source of vulnerabilities.

In my recent work on FreeBSD, I ran into the kernel crypto code.  It is worth mentioning that older crypto, particularly kernel crypto frameworks tend to hearken back to days when things were different than they are now.  For one, strong crypto was classified as a munition, and exporting it from various countries ran afoul of international arms trafficking laws.  Second, CPUs were much slower back then, crypto represented a more significant overhead, and most hardware crypto devices were attached to the PCI bus.  In the modern world, we have Bernstein v. United States (publication of crypto is protected free speech), CPUs are much faster, and hardware crypto typically takes the form of special CPU instructions, not devices that have to be accessed through kernel interfaces.

This state of affairs tends to lead to fairly fragmented crypto codebases, which is exactly what the FreeBSD kernel crypto codebase looks like.  Moreover, it completely lacks any public-key algorithms, which are necessary for kernel and driver signing.  Lastly, existing userland crypto libraries tend not to fair so well when being converted into kernel libraries, as they tend to rely on userland utilities to operate.

LiCl Goals

To address this, I recently started working on ideas for a lightweight, embeddable crypto library I’m calling LiCl.  The name of course is the chemical symbol for lithium chloride: a salt similar to sodium chloride (NaCl).  An interpretation of the name could be “lightweight interoperable crypto library”, though it occurred to me that “Lego-inspired crypto library” also works, as the design involves building cryptosystems out of “blocks”.

LiCl aims to produce a lightweight crypto library that is easy to use and also easy to drop into any application (userland, kernel, or embedded).  It has several design goals, which I’ll discuss here.

Control over Crypto through Policies

Aspects of the library should be governed by policies which can be set both at library build time as well as in any application that uses the library.  Policies should be as fine-grained as “don’t use these specific algorithms”, all the way up to things like “don’t use hardware random number generators”, or “only use safecurves-approved ECC”.  If done right, this also captures the configuration options necessary to say things like “don’t use anything that depends on POSIX userland”.

This is done in the implementation through a variety of C preprocessor definitions that control which implementations are present in the library, and which can be used by an application.

NaCl-Style Easy Interfaces

NaCl is designed to eliminate many bugs that can arise from improper use of crypto by providing the simplest possible interface through its “box” functions.  In NaCl, this works, as it aims to provide a crypto interface for network applications.

LiCl, on the other hand, aims to provide a more general toolbox.  Thus, it needs a way to build up a NaCl-style box out of components.  As we’ll see, I have a plan for this.

Curate Crypto, Don’t Implement It

Most of LiCl will be the code devoted to assembling the external crypto interfaces.  The actual crypto implementations themselves will be curated from various BSD-compatible licensed or public-domain source.  Now, of course, I may run into some algorithms that require direct implementation; however, I intend to actually write crypto code myself as a last resort.

Design Sketch

My plans for LiCl actually draw on programming language concepts to some degree, where objects describing components of a crypto scheme represent an AST-like structure that is used to generate a NaCl-style interface.  I’ll go into the various components I’ve worked out, and how they all fit together.

It should be remembered that this is a design in progress; I’m still considering alternatives.

The User-Facing Crypto Interfaces

Right now, there are six user-facing interfaces, all of which take the form of structs with function pointers, each of which take a user data pointer (in other words, the standard method for doing object-oriented programming in C).  The exact length of the user data depends on the components from which the interface was built.  The six interfaces are as follows:

• Symmetric-key encryption (stream cipher or block cipher with a mode)
• Symmetric-key authenticated encryption
• Symmetric-key authentication (hashing with a salt)
• Public-key encryption
• Public-key authenticated encryption (encryption with signature checking)
• Public-key authentication (signature verification)

These interfaces represent the combination of multiple crypto methods to create a complete package that should handle all the details in a secure fashion.  The result is that we can support encryption/decryption and signing/verification in a NaCl box-like interface.

Creating User-Facing Interfaces

A user creates one of the above interfaces by assembling components, each of which represents some cryptographic primitive or method (for example, a hash function, or a block cipher mode).  The key is ensuring that users assemble these components in a way that is valid and secure.  This will be guaranteed by a “build cryptosystem” function that performs a consistency check on the specification it’s given.  For example, it shouldn’t allow you to encrypt an authenticated message (encrypt-then-MAC).  Another reason for this is for supporting hardware crypto, which may impose various limits on how the primitives those implementations provide can be used.

I come from a programming language background, so I like to think about this in those terms.  The “build cryptosystem” function acts similarly to a compiler, and the rules are similar to a type system.  The key here is figuring out exactly what the “types” are.  This is an ongoing task, but it starts with figuring out what the basic component model looks like.  I have a good start on that, and have identified several kinds of components.

Components

Ultimately, we’ll build up a cryptosystem out of components.  A components is essentially a “block” of crypto functionality, which itself may be built out of other components.  For example, a keystore may require a random source.  I’ve sketched a list of components so far, and will discuss each one here:

Random Sources

Random sources are essential in any cryptosystem.  In LiCl, I want to support an HSM-style interface for key generation and storage, so it’s necessary to provide a random source for generating keys.  There are also concerns such as padding that require random bits.  Random sources are the only thing in the GitHub repo at the moment, and the only one is the POSIX urandom source.  The first curation task is to identify a high-quality software random number generator implementation that’s BSD/MIT licensed or public domain.

Keystores

LiCl’s interfaces are built around an assumption that there’s a layer of indirection between keys and their representation in memory.  This is done to enable use of HSMs and other hardware crypto.  A keystore interface represents such an indirection.

Keystores have a notion of an external representation for keys.  In the “direct” keystore implementation, this is the same as the actual representation; in an HSM-based keystore, it might be an ID number.  Keystores provide the ability to generate keys internally, add keys to the store, delete keys, and extract a key given its external representation.

The only implementation so far is the “direct” keystore, which is just a passthrough interface.  It requires a random source for its keygen functionality.

Arbitrary-Size Operations

One major building block is the ability to perform a given operation on arbitrary-sized data.  This is innate in some primitives, such as stream ciphers and hash functions.  In others, it involves things like modes of operation and padding.

This is where the type-like aspects begin to become visible.  For example, the GCM block cipher mode takes a fixed-size symmetric-key encryption operation and produces an arbitrary-sized symmetric-key authenticated encryption operation.  We could write this down in a semi-formal notation as “symmetric enc fixed size (n) -> symmetric auth enc variable block size(n)”.  Padding operations would eliminate the restriction on input size, and could be written as “algo variable block size (n), randsrc -> algo output variable output block size (n)”.

Of course, we wouldn’t write down this notation anywhere in the actual implementation (except maybe in the documentation).  In the code, it would all be represented as data structures.

Ultimately, we’d need to assemble components to get an arbitrary-sized operation with no input block size restriction.  We’d also need to match the algorithm type of the scheme we’re trying to build (so if we want authenticated symmetric key encryption, we need to ensure that’s what we build).

MAC Schemes and Signing

MAC schemes and signing algorithms both take a hash function and an encryption scheme and produce an authenticated encryption scheme.  Signing algorithms also require a public-key crypto scheme.  In the semi-formal notation, a MAC scheme might look something like this: “symmetric enc variable, hash -> symmetric auth enc variable”

Ciphers and Hashes

Ciphers are of course the basic building blocks of all this.  Ciphers may have different characteristics.  Block ciphers might be written as “symmetric enc fixed size(n)”.  An authenticated stream cipher would be written as “symmetric auth enc variable”.

Putting it All Together

Ultimately, the “build cryptosystem” functions will take a tree-like structure as an argument that describes how to combine all the various components to build a cryptosystem.  They then perform a consistency check on the whole tree to ensure that everything is put together correctly and then fill up a cryptosystem structure with all the implementation functions and data necessary to make it work.

Going Forward

With the design I’ve described, it should be possible to build a crypto library that will serve the needs of kernel and systems developer, but will also make it easier to use crypto in a manner that is correct.

The biggest remaining question is whether this design can effectively deal with the legacy interfaces that kernel developers must deal with.  However, it seems at least plausible that the model of assembling components should be able to handle this.  Afterall, even legacy systems are ultimately just assembling crypto primitives in a particular way; if a particular system can’t be modeled by the components LiCl provides, it should be possible to implement new components within the model I’ve described.

The repository for the project is here, however, there isn’t much there at this point.

FreeBSD EFI GELI Support

I have completed my work to add support for the GELI disk encryption system to the FreeBSD EFI boot loader.  This work started off intending to be a “simple” patch, but it became a much larger undertaking that ended up refactoring a significant portion of the EFI boot loader.

Regardless, the changeset is now usable and ready for testing.  It can be accessed on my GitHub.  I will be merging this periodically with the FreeBSD master in order to keep it up to date.

I am not recommending this for inclusion in the 11 release; it’s too big a change to incorporate this late in the game.

Design/Implementation Notes

This work breaks down into roughly four different components: EFI refactoring, boot crypto framework, GELI support, and kernel key injection.  I’ll cover each of these in turn.

EFI Refactoring

I have already written extensively about my EFI refactoring here.  The reason for undertaking this effort, however, was driven by GELI support.  Early in my work on this, I had implemented a non-EFI “providers” framework in boot1 in order to support the notion of disk partitions that may contain sub-partitions.  This was deeply unsatisfying to me for several reasons:

• It implemented a lot of the same functionality that exists in the EFI framework.
• It involved implementing a GPT partition driver to deal with partition tables inside GELI partitions (GPT detection and support is guaranteed by the EFI spec).
• The interface between the EFI framework and the custom “providers” framework was awkward.
• The driver was completely boot1-specific, and exporting it to something like GRUB probably involved a total rewrite.
• Implementing it within loader was going to involve a lot of code duplication.
• There was no obvious was to pass keys between boot1, loader, and the kernel.

Doing the EFIzation work eliminated these problems, and in my opinion, cleaned things up in boot1 as well.  The results were pleasing:

• The GELI driver can be extracted from the FreeBSD codebase without too much trouble.
• While I was unable to go all the way to the EFI driver model, the only blocker is the bcache code, and once that is resolved, we can have hotplug support in the boot loader!
• The boot1 and loader codebases are now sharing all the backend drivers, and boot1 has been reduced to one very small source file.

As I previously mentioned, my only reservation about this is increased dependence on (historically flaky) vendor-specific BIOS code.  However, with things like CoreBoot on the rise, I’m less concerned about this.

There are a couple of open questions and future work items coming out of this refactoring:

• Might it be a good idea to move the backend drivers out of boot and loader completely, and have them be wholly-separate EFI drivers that get installed by boot/loader?
• Provide EFI drivers for GELI and FreeBSD filesystems to projects like CoreBoot, TianoCore, and GRUB.
• Refactor the bcache code to support dynamic device detection and complete the transition to the EFI driver model.
• Play with possibilities that arise from hot-plugging, like pluggable USB dongles containing access credentials.

Boot Crypto

The boot crypto refactoring was a small effort to pull the crypto code out of the BIOS GELI code and put it in a common place for all boot utilities.  My hope is that boot_crypto becomes the go-to place for boot-time crypto code.

However, I’m a bit displeased with the state of crypto code in general.  I deemed it too ambitious to take that on in addition to everything else, but it seems that the crypto framework could profit from some work.  The problems I see are as follows:

• The codebase seems rather disorganized.  There are crypto and opencrypto in the kernel, and there doesn’t seem to be a common interface for all ciphers.
• There are insecure ciphers and algorithms (RC4, SHA1, MD5, DES) as well as missing modern ciphers (ChaCha20-Poly1305, stronger RipeMD variants, etc).
• The procedure for linking against the crypto codebase is harder than it could be.

The core crypto code should be usable in userland, kernel, and boot environments, with the only difference being the ABI for which it is compiled (native for user and kernel, MSABI for EFI, 32-bit for x86 BIOS, etc).  It should be possible to have a single codebase that gets turned into static libraries for each ABI and a shared library for userland.

It might be worth an experiment to replace the current crypto code with something like NaCL (libsodium) and use the scheme I describe above.

GELI Driver

The GELI EFI driver is a straightforward EFI bus[0] driver that detects the presence of a GELI volume on a device handle and creates a new device handle bearing an EFI_BLOCK_IO_PROTOCOL interface that provides access to the encrypted data.  This involves getting ahold of a password or a key, which I manage through the KMS driver (actually, the GELI driver currently contains the code to ask for a password directly, though this could and probably should be moved into the KMS driver).

 

[0]: EFI considers any driver that creates new handles to be a “bus driver”, even if it is something like a partition driver.

Kernel Key Injection and Boot KMS

Passing keys between boot1, loader, and the kernel was one of the key challenges of this work.  I also wanted to do this in a way that could be extended to support hardware security modules with a minimum of effort.  The EFI_KMS_PROTOCOL interface provided a means to accomplish this.

I provided another EFI driver that implements a software-based EFI_KMS_PROTOCOL interface.  This allows boot services to register keys, look them up later, and ultimately have them injected into the kernel.  This code is a fairly simple key table manager internally, and uses the file metadata functionality in loader to communicate keys to the kernel.

The current implementation leaves the task of asking for passwords up to the individual drivers, but I am seriously considering bringing that into the KMS driver as well, under the auspices of the CreateKey function.

On the kernel side, I added an interface to the crypto framework that provides access to the key buffer provided to the kernel by the KMS driver.  This allows keys to be passed from boot to kernel in a safer manner than the environment variable currently used by the BIOS GELI implementation.  It also provides a generic interface going forward for accomplishing this task.

This also puts things in such a state that a hardware-based key management system could be integrated.  Because boot1 and loader use EFI_KMS_PROTOCOL, which was specifically designed for key management systems, it is easy to add support for a device that supports this.  On the kernel side, we would simply expect the hardware KMS to be initialized with the keys we need.

Using and Testing EFI GELI Support

The changeset should be usable for most people.  I have tested the core functionality and have successfully loaded and booted kernels from ZFS volumes inside a GELI partition.  I am not actively using this, however, as I want to participate in the tests of the drm2 4.6 update, and I don’t want to assume the risks involved in merging two experimental branches.

If you want to use or test the changeset, I strongly recommend the following procedure:

• Be aware of the risks: this is full-disk encryption, so a problem may result in your files becoming inaccessible by any means.
• Read and understand the basics of EFI and the FreeBSD boot architecture.
• Install an EFI Shell program on your EFI System Partition, know how to run it from your BIOS menu.
• Install boot1 to an alternate location (like /efi/boot/bootx64.tst).
• Install loader to /boot/loader.tst on your main partition (I have left the LOADER_PATH variable set to /boot/loader.tst to facilitate this kind of testing)
• Start the EFI shell and run /efi/boot/bootx64.tst
• Try creating an empty GELI partition, and verify that boot1 and loader detect it[1]
• Try creating a filesystem in the GELI partition, copy your /boot/ directory from your main partition, see if boot1 and loader can read files from it
• Try loading and booting a kernel from your GELI partition
• Once you have succeeded at these, back up all your files, try converting your main disk into a GELI partition, and see if you can boot from it.

 

[1]: The EFIzed loader (with GELI support) should work fine even when loaded by the old boot1 (or by a tool such as GRUB)

Looking Forward: Tamper-Resilience

This changeset adds the first of three major capabilities that are needed to realize a longer-term tamper-resilience that I would very much like to see be a part of FreeBSD.  The steps toward this are as follows:

1. Support full-disk encryption from boot, with the ESP being the only non-encrypted partition on the system[2]
2. Support EFI secure boot of boot1, combined with signature checking for loader, the kernel, and device drivers, with the ability to designate a machine-specific signing key to be used to generate the signatures
3. Support secure suspend-to-disk

These three features when combined and implemented on a ZFS-based system create a powerful tamper-resilience scheme.  Because secure boot is used with a machine-specific platform key, this key can be stored on the encrypted disk.  This ensures that only someone with the ability to decrypt the disk is able to create a boot1 program that will run on the machine.  This prevents anyone from tampering with boot1 and hijacking the boot process.  Moreover, with suspend-to-disk, the machine is only vulnerable to data exfiltration when it is in active use.  When suspended or powered-off, everything is encrypted and protected by the secure boot scheme.

Obviously, this is not perfect; anyone able to overwrite the firmware or mess with the hardware can hijack the secure boot process and tamper with boot1.  This is why I call it tamper-resilience as opposed to tamper-proofing.  However, this scheme guarantees that the OS does not open any vulnerabilities to an attacker who wants to tamper with the system.

 

[2]: With projects like CoreBoot, it may become possible to have the firmware itself load programs from an encrypted volume

FreeBSD EFI boot/loader Refactor

I have just completed (for some value of “complete”) a project to refactor the FreeBSD EFI boot and loader code.  This originally started out as an investigation of a possible avenue in my work on GELI full-disk encryption support for the EFI boot and loader, and grew into a project in its own right.

More generally, this fits into a bunch of work I’m pursuing or planning to pursue in order to increase the overall tamper-resistance of FreeBSD, but that’s another article.

Background

To properly explain all this, I need to briefly introduce both the FreeBSD boot and loader architecture as well as EFI.

FreeBSD Boot Architecture

When an operating system starts, something has to do the work of getting the kernel (and modules, and often other stuff) off the disk and into memory, setting everything up, and then actually starting it.  This is the boot loader.  Boot loaders are often in a somewhat awkward position: they need to do things like read filesystems, detect some devices, load configurations, and do setup, but they don’t have the usual support of the operating system to get it done.  Most notably, they are difficult to work with because if something goes wrong, there is very little in the way of recovery, debugging, or even logging.

Moreover, back in the old days of x86 BIOS, space was a major concern: the BIOS pulled in the first disk sector, meaning the program had to fit into less than 512 bytes.  Even once a larger program was loaded, you were still in 16-bit execution mode.

To deal with this, FreeBSD adopted a multi-stage approach.  The initial boot loader, called “boot”, had the sole purpose of pulling in a more featureful loader program, called “loader”.  In truth, boot consisted of two stages itself: the tiny boot block, and then a slightly more powerful program loaded from a designated part of the BSD disklabel.

The loader program is much more powerful, having a full suite of filesystem drivers, a shell, facilities for loading and unloading the kernel, and other things.  This two-phase architecture overcame the severe limitations of the x86 BIOS environment.  It also allowed the platform-specific boot details to be separated from both the loader program and the kernel.  This sort of niceness is the hallmark of a sound architectural choice.

Inside the loader program, the code uses a set of abstracted interfaces to talk about devices.  Devices are detected, bound to a device switch structure, and then filesystem modules provide a way to access the filesystems those devices contain.  Devices themselves are referred to by strings that identify the device switch managing them.  This abstraction allows loader to support a huge variety of configurations and platforms in a uniform way.

The Extensible Firmware Interface

In the mid-2000’s, the Extensible Firmware Interface started to replace BIOS as the boot environment on x86 platforms.  EFI is far more modern, featureful, abstracted, and easy to work with than the archaic, crufty, and often unstandardized or undocumented BIOS.  I’ve written boot loaders for both; EFI is pretty straightforward, where BIOS is a tarpit of nightmares.

One thing EFI does is remove the draconian constraints on the initial boot loader.  The firmware loads a specific file from a filesystem, rather than a single block from a disk.  The EFI spec guarantees support for the FAT32 filesystem and the GUID Partition Table, and individual platforms are free to support others.

Another thing EFI does is provide abstracted interfaces for things like device IO, filesystems, and many other things.  Devices- both concrete hardware and derived devices such as disk partitions and network filesystems are represented using “device handles”, which support various operational interfaces through “protocol interfaces”, and are named using “device paths”.  Moreover, vendors and operating systems authors alike are able to provide their own drivers through a driver binding interface, which can create new device handles or bind new protocol interfaces to existing ones.

FreeBSD Loader and EFI Similarities

The FreeBSD loader and the EFI framework do many of the same things, and they do them in similar ways most of the time.  Both have an abstracted representation of devices, interfaces for interacting with them, and a way of naming them.  In many ways, the FreeBSD loader framework is prescient in that it did many of the things that EFI ended up doing.

The one shortcoming of FreeBSD loader is in the lack of support for dynamic device detection, also known as “hotplugging”.  When FreeBSD’s boot architecture was created (circa 1994), hotplugging was extremely uncommon: most hardware expected to be connected permanently and remain connected for the duration of operation.  Hence, the architecture was designed around a model of one-time static detection of all devices, and the code evolved around that assumption.  Hot-plugging was added to the operating system itself, of course, but there was little need for it in the boot architecture.  When EFI was born (mid 2000’s), hot-pluggable devices were common, and so supporting them was an obvious design choice.

EFI does this through its driver binding module, where drivers register a set of callbacks that check whether a device is supported, and then attempt to attach to it.  When a device is disconnected, another callback is invoked to disconnect it.  FreeBSD’s loader, on the other hand, expects to detect all devices in a probing phase during its initialization.  It then sets up additional structure (most notably, its new bcache framework) based on the list of detected devices.  Some phases of detection may rely on earlier ones; for example, the ZFS driver may update some devices that were initially detected as block devices.

Refactoring Summary

As I mentioned, my work on this was originally a strategy for implementing GELI support.  A problem with the two-phase boot process is that it’s difficult to get information between the two phases, particularly in EFI, where all code is position-independent, no hard addresses are guaranteed, and components are expected to talk through abstract interfaces.  (In other words, it rules out the sort of hacks that the non-EFI loader uses!)  This is a problem for something like GELI, which has to ask for a password to unlock the filesystem (we don’t want to ask for a password multiple times).  Also, much of what I was having to implement for GELI with abstract devices and a GPT partition driver and such ended up mirroring things that already existed in the EFI framework.

I ended up refactoring the EFI boot and loader to make more use of the EFI framework, particularly its protocol interfaces.  The following is a summary of the changes:

• The boot and loader programs now look for instances of the EFI_SIMPLE_FILE_SYSTEM_PROTOCOL, and use that interface to load files.
• The filesystem backend code from loader was moved into a driver which does the same initialization as before, then attaches EFI_SIMPLE_FILE_SYSTEM_PROTOCOL interfaces to all device handles that host supported filesystems.
• This is accomplished through a pair of wrapper interfaces that translate EFI_SIMPLE_FILE_SYSTEM_PROTOCOL and the FreeBSD loader framework’s filesystem interface back and forth.
• I originally wanted to move all device probing and filesystem detection into the EFI driver model, where probing and detection would be done in callbacks.  However, this didn’t work primarily because the bcache framework is strongly coupled to the static detection way of doing things.
• Interfaces and device handles installed in boot can be used by loader without problems.  This provides a way to pass information between phases.
• The boot and loader programs can also make use of interfaces installed by other programs, such as GRUB, or custom interfaces provided by open-source firmware.
• The boot and loader programs now use the same filesystem backend code; the minimal versions used by boot have been discarded.
• Drivers for things like GELI, custom partition schemes, and similar things can work by creating new device nodes and attaching device paths and protocol interfaces to them.

I sent an email out to -hackers announcing the patch this morning, and I hope to get GELI support up and going in the very near future (the code is all there; I just need to plug it in to the EFI driver binding and get it building and running properly).

For anyone interested, the branch can be found here: https://github.com/emc2/freebsd/tree/efize

Librem 13 FreeBSD Port

When the Librem laptops were announced last year, I was quite excited and I ordered both the 15 and 13-inch models.  My 13-inch model arrived last week, and I have begun the process of porting FreeBSD to it.

I have to say, I am very excited to finally have a laptop from a fully-cooperative manufacturer, where I can get my hands on all the hardware specs and possibly even upstream fixes.  This is a very welcome boon after a decade of having to deal with flaky BIOS issues, black-box hardware, and other difficulties.

The Laptop

The physical laptop itself is very solid and rather light.  It doesn’t creak, and the lid stays put even better than a macbook.  My only complaints are that the camera/microphone and wireless kill-switches are unlabeled, and that ethernet cables tend to fall out of the drop-down port.  Aside from those minor issues, I’m quite pleased with the physical unit.

It’s hard to see the kill-switches in the photo below, but they are on the hinge under the screen.

My only other regret is that the dvorak keyboard option became available after I’d ordered mine.  Oh well; maybe I can sweet-talk them into swapping it for me at a conference 😉

It was also very nice to unpack a laptop without implicitly accepting a Microsoft license agreement by opening the box!

BIOS and FreeBSD Installation

The first thing I do when I get a new laptop is poke around in the BIOS menu (no photos yet).  The librem has a coreboot port, but I decided to get FreeBSD installed and check the system out a bit before diving into the art of flashing my BIOS, so I was looking at the proprietary American Megatrends BIOS menu.  Even still, I was pleased by the features it presented, most notably the ability to set up custom signing keys.  I am going to have to do some work on a signed FreeBSD boot and loader chain.

My FreeBSD installation went off without any serious issues.  I installed FreeBSD 11 from a bootable memstick option, setting up a pure-ZFS system.  I had ordered a 1TB spindle drive and a 250GB SSD.  I reserved 48GB of the SSD for swap (total of 64GB memory).  I then set up a ZFS pool with the spindle drive as the main storage, a 16GB intent log on the SSD, and the rest of the SSD as an L2ARC cache device.  (I will eventually set up the ZFS volume to make all writes synchronous, so as to really use the intent log.)  I realize some might consider ZFS on a laptop to be overkill; however, I have found it to be an extremely versatile and stable filesystem.  It is incredibly crash-resistant and corruption-resistant, and its snapshotting is invaluable for risky updates.  The transparent compression features are useful as well, and can effectively increase your available space by a sizable amount.  Lastly, I have used the ability to serialize and deserialize the entire filesystem more than once.

I did encounter one of the issues in this process: a sporadic boot-hang and USB timeout that I now strongly suspect to be a timing bug in the FreeBSD boot process.

FreeBSD did handle the hardware kill-switches rather well (I’ve heard reports of Linux kernel panicking from them).  Flipping them off causes some kernel messages about timeouts, but the bus re-initializes upon flipping them back on.  If you boot with them off, then flip them on, the kernel detects the hardware properly.

FreeBSD Setup

The first thing I do on a new FreeBSD system is grab the source tree and build world, followed by kernel customization.  I noticed that building Clang has gotten pretty slow these days (which doesn’t bother me too much; I’d rather the compiler have a lot of optimization machinery than not).

After that, I grabbed the latest ports tree and started building the usual suspects to test the system (also, to get to where I could test X11).  I also grabbed Jean-Sebastian’s Intel graphics patch to see if that driver worked with the Broadwell card.  Sadly, it didn’t.

Working Hardware

Most of the hardware Just Works™, which is nice.  I was particularly pleased that all the fn-key combinations work out-of-the-box.  I have never seen that happen with any other vendor.

The following is a list of the working hardware:

• The EFI boot/loader
• SD card reader (mmc driver)
• Realtek Ethernet (re driver)
• System management bus and CPU frequency/temperature (smb, smbus, ichsmb, coretemp, cpufreq drivers)
• Intel High-Def Audio (snd_hda driver), though I haven’t tested the microphone yet.  Also, plugging into the headphone jack properly switches to headphones from the speakers (I’ve seen that not work).
• Hard Drive and SSD (obviously)
• USB ports
• Bluetooth

Unfortunately, the Intel accelerated graphics drivers don’t support the Broadwell cards.  This will come eventually, but FreeBSD is in the midst of a graphics framework overhaul to better track the Linux drivers.  Looks like it’s going to be VESA for now.

Current Issues

There are currently some issues, which I will be working to fix:

• The Atheros 9462 card is detected, but the radio doesn’t seem to be working.  The pciconf tool reports a few errors, and scans seem to run, but don’t pick up anything.  I have confirmed this is not a hardware issue by booting with a Kali linux memstick.
• Blank screen on resume.  My initial investigations reveal some ACPI execution errors during resume, which may be related.  I need to get up in the kernel source and add some logging to see what’s going on.
• VESA wierdness with X11.  The VESA X driver works mostly, but if you switch back to the terminal, there’s a couple of pixels around the border of the screen that stay the way they looked in X.  Also, when you shutdown X, the screen freezes and the logs indicate some kind of timeout.  Both of these seem to implicate the VGA BIOS.
• Sporadic boot-hang and USB timeouts.  These seem to be specific to a kernel configuration, and go away when changing the verbosity level.  This strongly indicates a timing-related bug in the kernel initialization procedures.

Of these issues, the wireless card and blank screen are the most critical, followed by the X11 weirdness.  I will be in contact with the Librem developers should my initial attempts to fix these issues prove unsuccessful.

Following that, I want to see if there’s a way to make the kill-switches behave more gracefully.  If the USB driver could be connected to treat those devices as hot-pluggable, or else assume timeouts are disconnects.

In any case, stay tuned for updates…