Ubuntu 22.04 (or 21.10), kubernetes & cgroups v2

On some of our systems we've recently upgraded Ubuntu to 22.04 LTS (from 20.04 LTS). And one thing that has been problematic afterwards is that containers kept crashing - or to be more exact: the containers have been killed because the probes failed.

This affected cilium as well as the kube-apiserver or kube-controller. The most common denominator has been that all affected containers have been executed in hostNetwork mode.

The error message has been like:

Startup probe failed: Get "http://127.0.0.1:9879/healthz": dial tcp 127.0.0.1:9879: connect: connection refused

Validating the connection from the host itself showed it is working successfully.

The reason for this is that Ubuntu enabled cgroups v2 with 21.10 (or 22.04 LTS) by default and that this causes issues.

Workaround: disable cgroups v2

Until now we only have a workaround for this, by enabling cgroups v1 using a boot parameter. This can be done setting it like this:

GRUB_CMDLINE_LINUX="systemd.unified_cgroup_hierarchy=0"

Having applied this, update grub and reboot afterwards.

update-grub

Now you have a similar behaviour like on 20.04 LTS. Having said that, it's just a workaround, we're still checking out a solution that makes use uf cgroups v2.