Photo by Maxim Abramov / Unsplash

Sometimes things only work because AppArmor does not

Felix Zimmermann
Felix Zimmermann

Ubuntu Server ships with AppArmor, a Linux kernel security module that hardens hosts by restricting what each process may or may not do. This is driven by profiles. Processes without a dedicated profile typically run "unconfined". In a container runtime, this is different: Typically, the workload gets the profile of the container runtime, unless the operator overwrites this.
After an OS/kernel update and reboot, pods on a node failed to become ready or misbehaved. Most noticeably, the databases never became healthy. PostgreSQL for example logged:

could not signal for checkpoint: Permission denied

What actually broke

Ubuntu enables kernel.apparmor_restrict_unprivileged_unconfined=1 by default (docs). When unprivileged runc applies the container profile, that restriction turns the switch into a stack, so tasks run as cri-containerd.apparmor.d//&unconfined instead of plain cri-containerd.apparmor.d.

A kernel fix (LP: #2169638) then started enforcing that stacked label as a distinct peer. containerd’s default profile only allowed signals to peer=cri-containerd.apparmor.d — not the stacked name — so processes inside one container could no longer signal each other (containerd#14485). PostgreSQL’s checkpoint signalling was just the loudest failure.

How to fix this?

Workaround

If you need the node back immediately:

echo 'kernel.apparmor_restrict_unprivileged_unconfined = 0' | sudo tee /etc/sysctl.d/99-apparmor-unconfined.conf
sudo sysctl --system

Then recreate the affected pods. New containers get the plain cri-containerd.apparmor.d label again and intra-container signals work. This weakens a deliberate host hardening control, so treat it as temporary (same workaround discussed in containerd#14487 and related reports).

Permanent solution

containerd’s fix is to allow signal / ptrace for the stacked peer as well as the plain profile name (containerd/containerd#14487). Once that lands in the containerd (or k3s/RKE2-bundled containerd) build you run, recreate the pods so they pick up the updated profile. That keeps Ubuntu’s hardening sysctl enabled.