Unexpected behavior of TUN devices in Kubernetes >= 1.31.3
Sometimes, security improvements in one project can cause problems in places nobody ever expected. In this case, we had to deal with one of these improvements.
Some workloads, such as openVPN or FortiManager, need to work with the TUN-device /dev/net/tun. We have such workloads running kubernetes.
After updating to kubernetes v1.31.3 to mitigate a regression in kubernetes itself, the previously running application failed to start.
A look at the application logs gave a hint:
tun.c,377:Could not open /dev/net/tun.
failed to create tun device: No such file or directory
tunnel init failed
To connect to various other systems, the application creates a TUN device at container startup. For this to work, the pod needed the "NET_ADMIN" capability. But since the update this isn't true anymore. Even with the "ALL" capability set, the application failed to start.
Workaround
To solve this problem, starting with kubernetes 1.31.3 the pod must to be started in privileged mode. With this enabled, the TUN device can be created from inside the container and the application can access the TUN device as expected.
Background
The root cause of this problem is a change in runc (see the discussion here: https://github.com/opencontainers/runc/pull/3468) that was released with version 1.2.0. The maintainers decided that containers should not have access to the tun/tap device by default, and that you should explicitly grant access. This must be done by manually adding the device. Using runc this can be done via device entries. Using for example docker, this can be done via the --device parameter. Kubernetes itself has no way to add devices to a pod, so the pod has to be started as a privileged one.
Because of the impact of these changes, there is currently some discussion about rolling them back. But as of this writing, no decision has been made, so we'll have to wait and see what the final solution will be in the future.
