TIL swap turns Kubernetes OOM kills into zombie nodes

Sunday evening, three of the four Raspberry Pi nodes in my k3s cluster went NotReady at once. SSH? Connection reset by peer. The nodes weren’t dead — they just wouldn’t talk to anyone. One of them only came back after a power cycle.

The culprit: swap. Raspberry Pi OS ships with dphys-swapfile enabled, a swapfile sitting on the SD card. A stock kubelet refuses to start while swap is on (--fail-swap-on defaults to true, and kubeadm’s preflight checks make you disable it), but k3s sets fail-swap-on=false for you. So a k3s cluster on Pis runs on swap from day one, and nothing ever tells you.

Kubernetes wants swap off because its whole memory model assumes it. When a node runs out of RAM, the kubelet evicts pods or the kernel OOM-kills them, and they get restarted elsewhere. Ugly, but fast — and self-healing. With swap, the kernel instead does you the “favor” of paging memory out to storage. On an SD card, that means the node grinds into a thrash loop: nothing gets killed, nothing recovers, everything just gets slower until the kubelet misses its lease renewals (hello NotReady) and sshd can’t even finish a handshake. Swap converts a fast, recoverable pod failure into a slow node death that ends at the power outlet.

Two more things I learned while cleaning up:

swapoff on a struggling node looks hung. It isn’t — it has to page everything back into RAM first, and pulling that off an SD card under memory pressure takes minutes. Watch it instead of killing it:

watch 'cat /proc/swaps; echo; free -h'

The permanent fix on Raspberry Pi OS:

sudo dphys-swapfile swapoff
sudo systemctl disable --now dphys-swapfile

Postscript: since Kubernetes 1.34, swap is actually supported — NodeSwap went GA with its LimitedSwap mode for burstable pods. But that’s a deliberate, configured trade-off, not a distro quietly leaving a swapfile on your SD card.