Ubuntu 20.04 Root on ZFS :: TODO: Document — typealias Writer = Developer

NOTE: This guide is adapted from the OpenZFS project Wiki on installing Ubuntu 18.04 Root on ZFS. In some of the steps there are very specific parameters that are specific to my build and my environment. This post is meant to document my configuration.

Step 1: Prepare The Install Environment⌗

1.1 Boot the Ubuntu Live CD. Select Try Ubuntu. Connect your system to the Internet as appropriate (e.g. join your WiFi network). Open a terminal (press Ctrl-Alt-T).

1.2 Setup and update the repositories:

sudo apt-add-repository universe
sudo apt update

1.3 Become root:

sudo -i

1.4 Install ZFS in the Live CD environment:

apt install --yes debootstrap gdisk zfs-initramfs

Step 2: Disk Formatting⌗

2.1 Set a variable with the disk name:

DISK=/dev/disk/by-id/ata-ADATA_SU800_2J3920158860

Always use the long /dev/disk/by-id/* aliases with ZFS. Using the /dev/sd* device nodes directly can cause sporadic import failures, especially on systems that have more than one storage pool.

2.2 Partition your disk(s):

Clear the partition table:

wipefs -a $DISK
sgdisk --zap-all $DISK

Create the UEFI /boot/efi partition:

sgdisk     -n2:1M:+1G   -t2:EF00 $DISK
mkfs.fat -F32 ${DISK}-part2

Create the EXT4 /boot partition

sgdisk     -n3:0:+64G   -t3:8300 $DISK
mkfs.ext4 ${DISK}-part3

Create the Swap partition

sgdisk     -n4:0:0      -t4:8200 $DISK
mkswap ${DISK}-part4

2.5 Create the root pool:

zpool create -o ashift=12 \
    -O acltype=posixacl -O canmount=off -O compression=lz4 \
    -O dnodesize=auto -O normalization=formD -O relatime=on -O xattr=sa \
    -O mountpoint=/ -R /mnt pond \
    mirror /dev/disk/by-id/ata-WDC_WD40EFRX-68N32N0_WD-WCC7K3HN32XE /dev/disk/by-id/ata-WDC_WD40EFRX-68N32N0_WD-WCC7K0AUHVN0 \
    mirror /dev/disk/by-id/ata-WDC_WD40EFRX-68N32N0_WD-WCC7K2VLLA6H /dev/disk/by-id/ata-WDC_WD40EFRX-68N32N0_WD-WCC7K5VRK3S8 \
    mirror /dev/disk/by-id/ata-WDC_WD20EFRX-68EUZN0_WD-WCC4M3RY4D7N /dev/disk/by-id/ata-WDC_WD20EFRX-68EUZN0_WD-WCC4M3EKYP41 \
    mirror /dev/disk/by-id/ata-WDC_WD20EFRX-68EUZN0_WD-WCC4M4TJ1F2V /dev/disk/by-id/ata-WDC_WD20EFRX-68EUZN0_WD-WCC4M4VDJ1P6 \
    log mirror /dev/disk/by-id/ata-ADATA_SU800_2H2120013305 /dev/disk/by-id/ata-ADATA_SU800_2J3920157929

The use of ashift=12 is recommended here because many drives today have 4KiB (or larger) physical sectors, even though they present 512B logical sectors. Also, a future replacement drive may have 4KiB physical sectors (in which case ashift=12 is desirable) or 4KiB logical sectors (in which case ashift=12 is required).
Setting -O acltype=posixacl enables POSIX ACLs globally. If you do not want this, remove that option, but later add -o acltype=posixacl (note: lowercase “o”) to the zfs create for /var/log, as journald requires ACLs
Setting normalization=formD eliminates some corner cases relating to UTF-8 filename normalization. It also implies utf8only=on, which means that only UTF-8 filenames are allowed. If you care to support non-UTF-8 filenames, do not use this option. For a discussion of why requiring UTF-8 filenames may be a bad idea, see The problems with enforced UTF-8 only filenames.
Setting relatime=on is a middle ground between classic POSIX atime behavior (with its significant performance impact) and atime=off (which provides the best performance by completely disabling atime updates). Since Linux 2.6.30, relatime has been the default for other filesystems. See RedHat’s documentation for further information.
Setting xattr=sa vastly improves the performance of extended attributes. Inside ZFS, extended attributes are used to implement POSIX ACLs. Extended attributes can also be used by user-space applications. They are used by some desktop GUI applications. They can be used by Samba to store Windows ACLs and DOS attributes; they are required for a Samba Active Directory domain controller. Note that xattr=sa is Linux-specific. If you move your xattr=sa pool to another OpenZFS implementation besides ZFS-on-Linux, extended attributes will not be readable (though your data will be). If portability of extended attributes is important to you, omit the -O xattr=sa above. Even if you do not want xattr=sa for the whole pool, it is probably fine to use it for /var/log.

Hints:

The pool name is arbitrary. If changed, the new name must be used consistently. On systems that can automatically install to ZFS, the root pool is named rpool by default.

Step 3: System Installation⌗

3.1 Create filesystem datasets to act as containers:

zfs create -o canmount=off -o mountpoint=none pond/ROOT

On Solaris systems, the root filesystem is cloned and the suffix is incremented for major system changes through pkg image-update or beadm. Similar functionality for APT is possible but currently unimplemented. Even without such a tool, it can still be used for manually created clones.

3.2 Create filesystem datasets for the root and boot filesystems:

zfs create -o canmount=on -o mountpoint=/ pond/ROOT/ubuntu

3.3 Create datasets:

zfs create                                 pond/home
zfs create -o mountpoint=/root             pond/home/root
zfs create -o canmount=off                 pond/var
zfs create -o canmount=off                 pond/var/lib
zfs create                                 pond/var/log
zfs create                                 pond/var/spool
zfs create -o com.sun:auto-snapshot=false  pond/var/cache
zfs create -o com.sun:auto-snapshot=false  pond/var/tmp
chmod 1777 /mnt/var/tmp
zfs create                                 pond/opt
zfs create                                 pond/srv
zfs create -o canmount=off                 pond/usr
zfs create                                 pond/usr/local
zfs create                                 pond/var/www
zfs create -o com.sun:auto-snapshot=false  pond/var/lib/docker
zfs create -o com.sun:auto-snapshot=false  pond/var/lib/nfs

If you do nothing extra, /tmp will be stored as part of the root filesystem. Alternatively, you can create a separate dataset for /tmp, as shown above. This keeps the /tmp data out of snapshots of your root filesystem. It also allows you to set a quota on rpool/tmp, if you want to limit the maximum space used. Otherwise, you can use a tmpfs (RAM filesystem) later.

zfs create -o setuid=off -o devices=off -o sync=disabled -o com.sun:auto-snapshot=false pond/tmp
chmod 1777 /mnt/tmp

The primary goal of this dataset layout is to separate the OS from user data. This allows the root filesystem to be rolled back without rolling back user data such as logs (in /var/log). This will be especially important if/when a beadm or similar utility is integrated. The com.sun.auto-snapshot setting is used by some ZFS snapshot utilities to exclude transient data.

3.4 Install the minimal system:

debootstrap focal /mnt http://archive.ubuntu.com/ubuntu
zfs set devices=off pond

The debootstrap command leaves the new system in an unconfigured state. An alternative to using debootstrap is to copy the entirety of a working system into the new ZFS root.

Step 4: System Configuration⌗

4.1 Configure the hostname

echo the-clubhouse.local > /mnt/etc/hostname

vi /mnt/etc/hosts
Add a line:
127.0.1.1       the-clubhouse.local

4.2 Configure the network interface:

vi /mnt/etc/netplan/01-netcfg.yaml
network:
  version: 2
  ethernets:
    enp8s0:
      dhcp4: true
      dhcp6: true
    enp12s0:
      dhcp4: true
      dhcp6: true

4.3 Configure the package sources:

vi /mnt/etc/apt/sources.list
deb http://archive.ubuntu.com/ubuntu focal main universe
deb-src http://archive.ubuntu.com/ubuntu focal main universe

deb http://security.ubuntu.com/ubuntu focal-security main universe
deb-src http://security.ubuntu.com/ubuntu focal-security main universe

deb http://archive.ubuntu.com/ubuntu focal-updates main universe
deb-src http://archive.ubuntu.com/ubuntu focal-updates main universe

4.4 Bind the virtual filesystems from the LiveCD environment to the new system and chroot into it:

mount --rbind /dev  /mnt/dev
mount --rbind /proc /mnt/proc
mount --rbind /sys  /mnt/sys
chroot /mnt /usr/bin/env DISK=$DISK bash --login

Note: This is using --rbind, not --bind.

4.5 Configure a basic system environment:

apt update

dpkg-reconfigure locales
dpkg-reconfigure tzdata

4.6 Install ZFS in the chroot environment for the new system:

apt install --yes --no-install-recommends linux-image-generic
apt install --yes zfs-initramfs

4.7 Mount /boot and /boot/efi

mkdir /boot
echo PARTUUID=$(blkid -s PARTUUID -o value ${DISK}-part3) \
    /boot ext4 errors=remount-ro 0 1 >> /etc/fstab
echo PARTUUID=$(blkid -s PARTUUID -o value ${DISK}-part2) \
    /boot/efi vfat nofail,x-systemd.device-timeout=1 0 1 >> /etc/fstab
mount /boot
mkdir /boot/efi
mount /boot/efi
apt install --yes grub-efi-amd64-signed shim-signed

4.8 Set a root password

passwd

4.9 zfs-mount-generator

mkdir -p /etc/zfs/zfs-list.cache
touch /etc/zfs/zfs-list.cache/pond
PROPS="name,mountpoint,canmount,atime,relatime,devices,exec,readonly"
PROPS="${PROPS},setuid,nbmand,encroot,keylocation"
zfs list -H -t filesystem -o $PROPS -r pond > /etc/zfs/zfs-list.cache/pond
:%s/mnt\/*//
ln -s /usr/lib/zfs-linux/zed.d/history_event-zfs-list-cacher.sh /etc/zfs/zed.d
#systemctl enable zfs-import-cache
systemctl enable zfs-import.target
systemctl enable zfs-zed.service
systemctl enable zfs.target

Hint: See man 8 zfs-mount-generator, e.g., man zfs-mount-generator, and follow instructions there if these instructions are incorrect.

4.10 Setup system groups:

addgroup --system lpadmin
addgroup --system sambashare

Step 5: GRUB Installation⌗

5.1 Verify that the /boot filesystem is recognized:

grub-probe /boot

5.2 Refresh the initrd files:

update-initramfs -u -k all

5.3 Workaround GRUB’s missing zpool-features support:

vi /etc/default/grub
Set: GRUB_CMDLINE_LINUX="root=ZFS=rpool/ROOT/ubuntu"

5.4 Make debugging GRUB easier:

vi /etc/default/grub
Comment out: GRUB_TIMEOUT_STYLE=hidden
Set: GRUB_TIMEOUT=5
Below GRUB_TIMEOUT, add: GRUB_RECORDFAIL_TIMEOUT=5
Remove quiet and splash from: GRUB_CMDLINE_LINUX_DEFAULT
Uncomment: GRUB_TERMINAL=console
Save and quit.

5.5 Update the boot configuration:

update-grub

Note: Ignore errors from osprober, if present.

5.6 Install the boot loader

grub-install --target=x86_64-efi --efi-directory=/boot/efi \
    --bootloader-id=ubuntu --recheck --no-floppy

5.7 Verify that the ZFS module is installed:

ls /boot/grub/*/zfs.mod

5.8 Unmount the /boot and /boot/efi partitions

umount /boot/efi
umount /boot

5.9 Prevent tmpfs from being mounted:

systemctl mask tmp.mount

Step 6: First Boot⌗

6.1 Snapshot the initial installation:

zfs snapshot pond/ROOT/ubuntu@install

In the future, you will likely want to take snapshots before each upgrade, and remove old snapshots (including this one) at some point to save space.

6.2 Exit from the chroot environment back to the LiveCD environment:

exit

6.3 Run these commands in the LiveCD environment to unmount all filesystems:

mount | grep -v zfs | tac | awk '/\/mnt/ {print $3}' | xargs -i{} umount -lf {}
zpool export -a

6.4 Reboot:

reboot

6.5 Wait for the newly installed system to boot normally. Login as root.

6.6 Create a user account:

zfs create pond/home/rlovelett
adduser --home /home/rlovelett --shell /bin/zsh --no-create-home --uid 1000 --gecos "Ryan Lovelett" rlovelett
zfs create pond/home/llovelett
adduser --home /home/llovelett --no-create-home --uid 1001 --gecos "Lindsey Lovelett" llovelett

6.7 Add your user account to the default set of groups for an administrator:

usermod -a -G adm,cdrom,dip,lpadmin,plugdev,sambashare,sudo rlovelett
usermod -a -G adm,cdrom,dip,lpadmin,plugdev,sambashare,sudo llovelett

Step 7: Configure Swap⌗

echo PARTUUID=$(blkid -s PARTUUID -o value ${DISK}-part4) \
    none swap sw 0 0 >> /etc/fstab
swapon -av

Step 8: Full Software Installation⌗

8.1 Upgrade the minimal system:

apt dist-upgrade --yes

8.2 Disable log compression:

As /var/log is already compressed by ZFS, logrotate’s compression is going to burn CPU and disk I/O for (in most cases) very little gain. Also, if you are making snapshots of /var/log, logrotate’s compression will actually waste space, as the uncompressed data will live on in the snapshot. You can edit the files in /etc/logrotate.d by hand to comment out compress, or use this loop (copy-and-paste highly recommended):

for file in /etc/logrotate.d/* ; do
    if grep -Eq "(^|[^#y])compress" "$file" ; then
        sed -i -r "s/(^|[^#y])(compress)/\1#\2/" "$file"
    fi
done

8.4 Reboot:

reboot

Step 9: Final Cleanup⌗

9.1 Wait for the system to boot normally. Login using the account you created. Ensure the system (including networking) works normally.

9.2 Disable the root password

sudo usermod -p '*' root