Swap turns Kubernetes OOM kills into zombie nodes
Three of the four Raspberry Pi nodes in my k3s cluster (k3s is a lightweight Kubernetes distribution) went NotReady at once one Sunday evening. SSH gave me Connection reset by peer. The nodes weren’t dead, they just wouldn’t talk to anyone. (One of them only came back after a power cycle.)
The cause was swap. Raspberry Pi OS ships with dphys-swapfile enabled, a swapfile on the SD card. A stock kubelet refuses to start while swap is on (--fail-swap-on defaults to true, and kubeadm’s preflight checks make you disable it). k3s sets fail-swap-on=false for you, though. So a k3s cluster on Pis runs on swap from day one, and nothing ever tells you.
Kubernetes wants swap off because its memory model assumes there is none. When a node runs out of RAM, the kubelet evicts pods or the kernel OOM-kills them, and they get restarted elsewhere. That’s ugly, but fast, and it heals itself. With swap, the kernel does you the “favor” of paging memory out to storage instead. On an SD card the node grinds into a thrash loop, where nothing gets killed, nothing recovers and everything just gets slower. Then the kubelet misses its lease renewals (hello NotReady) and sshd can’t even finish a handshake. So with swap, a pod failure that recovers fast becomes a slow node failure that only ends at the power outlet.
Two more things I learned while cleaning up:
swapoff on a struggling node looks hung, but it’s still working. It has to page everything back into RAM first, and pulling that off an SD card under memory pressure takes minutes. Watch it instead of killing it:
watch 'cat /proc/swaps; echo; free -h'
The permanent fix on Raspberry Pi OS:
sudo dphys-swapfile swapoff
sudo systemctl disable --now dphys-swapfile
Since Kubernetes 1.34, swap is supported. NodeSwap went GA with its LimitedSwap mode for burstable pods. That’s something you configure on purpose, though. My problem was a swapfile the distro enables by default, which k3s then runs on without a warning.