Operating System

For this setup Fedora is the Operating System of choice for multiple reasons. It provides both image-based installation methods (for example Fedora CoreOS) as well as package-based installation methods (for example Fedora Server) for many architectures and provides a modern and stable set of packages. Further it matches the developer machine OS, which helps with debugging and testing things locally before pushing them onto the deployments.

OS requirements

The OS requirements to run the current setup are:

  • modern software versions
  • Kubeadm support
  • cri-o support
  • TPM-based LUKS encryption
  • SELinux support
  • (optional) cockpit integration
  • (optional) SSH access
  • automated updates

Setup script

Currently the following script is used for set up:


# System upgrade
dnf upgrade -y

# Install cri-o and kubernetes
dnf copr enable -y "sheogorath/kubernetes-1.28"
dnf install -y cri-o cri-tools kubernetes kubernetes-kubeadm
systemctl enable --now crio

# Load kernel modules for Kubernetes and Calico
modprobe br_netfilter
modprobe wireguard
cat <<EOF | sudo tee /etc/modules-load.d/k8s.conf

# Prepare sysctls for Kubernetes
cat <<EOF | sudo tee /etc/sysctl.d/k8s.conf
net.bridge.bridge-nf-call-ip6tables = 1
net.bridge.bridge-nf-call-iptables = 1
net.ipv4.ip_forward = 1
sysctl --system

dnf install -y iptables

# Disable systemd-resolved for CoreDNS
rm -f /etc/resolv.conf
cp /run/systemd/resolve/resolv.conf /etc/resolv.conf
systemctl disable --now systemd-resolved

# Prepare NetworkManager for Calico
cat <<EOF | sudo tee /etc/NetworkManager/conf.d/calico.conf
systemctl restart NetworkManager

systemctl mask firewalld

# Disable zram swap
dnf remove -y zram-generator-defaults

# Setup TPM encryption
dnf install -y clevis-dracut
clevis luks bind -d /dev/nvme0n1p3 tpm2 '{}'
dracut -f


Be aware that this interactive due to TPM set up

Filesystem Layout

/xfs50GiBRoot filesystem set up by Fedora Server layout.
/boot/efivfat600MiBFilesystem for EFI, set up by Fedora Server layout.
/var/lib/containersxfs50GiBFilesystem for container images.
/var/lib/kubeletxfs20GiBFilesystem for kubelet related storage, such as emptyDir
/var/lib/longhornxfsvariesFilesystem for longhorn storage, this is used by longhorn to provide high-available storage across the clusters.
/var/lib/storagexfsvariesAdditional filesystem for longhorn storage, this is used by longhorn to provide high-available storage across the clusters.

Setup addition SSD

# Setup LUKS recovery key
cryptsetup luksFormat /dev/sda
cryptsetup isLuks /dev/sda
cryptsetup luksDump /dev/sda
cryptsetup luksUUID /dev/sda
cryptsetup luksOpen /dev/sda storage
# Encrypt with local TPM
clevis luks bind -d /dev/sda tpm2 '{}'
mkfs.xfs /dev/mapper/storage
echo "storage UUID=$(cryptsetup luksUUID /dev/sda) none discard,timeout=15" >> /etc/crypttab
echo "/dev/mapper/storage   /var/lib/storage           xfs     defaults,x-systemd.device-timeout=0 0 0" >> /etc/fstab
mkdir -p /var/lib/storage
chcon -t container_file_t /var/lib/storage/
mount -a
df -h /var/lib/storage/
# Make sure decryption on reboot works
systemctl enable clevis-luks-askpass.path