Operating System
For this setup Fedora is the Operating System of choice for multiple reasons. It provides both image-based installation methods (for example Fedora CoreOS) as well as package-based installation methods (for example Fedora Server) for many architectures and provides a modern and stable set of packages. Further it matches the developer machine OS, which helps with debugging and testing things locally before pushing them onto the deployments.
OS requirements
The OS requirements to run the current setup are:
- modern software versions
- Kubeadm support
- cri-o support
- TPM-based LUKS encryption
- SELinux support
- (optional) cockpit integration
- (optional) SSH access
- automated updates
Setup script
Currently the following script is used for set up:
#!/bin/bash
# System upgrade
dnf upgrade -y
# Install cri-o and kubernetes
dnf copr enable -y "sheogorath/kubernetes-1.28"
dnf install -y cri-o cri-tools kubernetes kubernetes-kubeadm
systemctl enable --now crio
# Load kernel modules for Kubernetes and Calico
modprobe br_netfilter
modprobe wireguard
cat <<EOF | sudo tee /etc/modules-load.d/k8s.conf
br_netfilter
wireguard
EOF
# Prepare sysctls for Kubernetes
cat <<EOF | sudo tee /etc/sysctl.d/k8s.conf
net.bridge.bridge-nf-call-ip6tables = 1
net.bridge.bridge-nf-call-iptables = 1
net.ipv4.ip_forward = 1
EOF
sysctl --system
dnf install -y iptables
# Disable systemd-resolved for CoreDNS
rm -f /etc/resolv.conf
cp /run/systemd/resolve/resolv.conf /etc/resolv.conf
systemctl disable --now systemd-resolved
# Prepare NetworkManager for Calico
cat <<EOF | sudo tee /etc/NetworkManager/conf.d/calico.conf
[keyfile]
unmanaged-devices=interface-name:cali*;interface-name:tunl*;interface-name:vxlan.calico;interface-name:wireguard.cali
EOF
systemctl restart NetworkManager
systemctl mask firewalld
# Disable zram swap
dnf remove -y zram-generator-defaults
# Setup TPM encryption
dnf install -y clevis-dracut
clevis luks bind -d /dev/nvme0n1p3 tpm2 '{}'
dracut -f
reboot
Be aware that this interactive due to TPM set up
Filesystem Layout
| Path | Filesystem | Size | Description |
|---|---|---|---|
/ | xfs | 50GiB | Root filesystem set up by Fedora Server layout. |
/boot/efi | vfat | 600MiB | Filesystem for EFI, set up by Fedora Server layout. |
/var/lib/containers | xfs | 50GiB | Filesystem for container images. |
/var/lib/kubelet | xfs | 20GiB | Filesystem for kubelet related storage, such as emptyDir |
/var/lib/longhorn | xfs | varies | Filesystem for longhorn storage, this is used by longhorn to provide high-available storage across the clusters. |
/var/lib/storage | xfs | varies | Additional filesystem for longhorn storage, this is used by longhorn to provide high-available storage across the clusters. |
Setup addition SSD
# Setup LUKS recovery key
cryptsetup luksFormat /dev/sda
cryptsetup isLuks /dev/sda
cryptsetup luksDump /dev/sda
cryptsetup luksUUID /dev/sda
cryptsetup luksOpen /dev/sda storage
# Encrypt with local TPM
clevis luks bind -d /dev/sda tpm2 '{}'
mkfs.xfs /dev/mapper/storage
echo "storage UUID=$(cryptsetup luksUUID /dev/sda) none discard,timeout=15" >> /etc/crypttab
echo "/dev/mapper/storage /var/lib/storage xfs defaults,x-systemd.device-timeout=0 0 0" >> /etc/fstab
mkdir -p /var/lib/storage
chcon -t container_file_t /var/lib/storage/
mount -a
df -h /var/lib/storage/
# Make sure decryption on reboot works
systemctl enable clevis-luks-askpass.path