Get filesystem access using an ephemeral container in kubernetes

Kubernetes has a feature called ephemeral containers since 1.25 in stable state. Ephemeral containers are - as the name states - temporary containers that might be attached to existing pods.

Why ephemeral containers are awesome

Ephemeral containers help to attach a container to an already running pod. The typical use case for this might be that you are a software engineer and need to diagnose an issue on a running instance that cannot be restarted (because this might result in loss of the errorneous state of the application).

Ephemeral containers allow you to attach an additional container with a specified image to the same process namespace as another container in a given pod.

Being attached to the namespace allows you to interact with the applications and perform for example a memory dump or even attach a debugger to the running app.

But how to access the filesystem?

When you are running a given container you might not neccessarily need to interact with the processes itself but you might want to interact with files within such a container. A rather simple example is that if you need to extract some files from a container that is built distroless or has just no tar installed (which is required for kubectl cp), an ephemeral container comes in handy.

Luckily there's a way to achieve this.

1. Add an ephemeral container to the pod and container you want to access
2. Switch to the uid that matches the uid running inside the target container
3. Traverse to the target container via /proc/<id>/root

As a sequence of command it looks like

# enter ephemeral container
kubectl debug app-to-debug --target=app --image=ubuntu:latest -it --tty=true -- bash

# make sure target user has a shell
# in this case we want to switch to www-data
usermod -s /bin/bash www-data
su www-data

# navigate to the target
cd /proc/1/root

Now you can access the target filesystem as the same user that is running the target container.

In one of my next posts I will describe another approach that allows you to access and modify contents of the volumes of a container you might want to debug.