You've successfully subscribed to Nuvotex Blog
Great! Next, complete checkout for full access to Nuvotex Blog
Welcome back! You've successfully signed in.
Success! Your account is fully activated, you now have access to all content.
Success! Your billing info is updated.
Billing info update failed.

Kubernetes, blockDevices & denied permissions

Mounting (block)devices on containers might result in permission errors (Permission denied) if pods have applied a securityContext. It's possible to configure this on containerd and get both of best worlds - fast device access and reduced permissions on containers.

Daniel Nachtrub
Daniel Nachtrub

We're currently quite diving into kubevirt and came upon an issue with block volumes that have been cloned using datavolume.

The datavolume looks like this

kind: DataVolume
  name: debian-vm-clone1-os-disk
    volumeMode: Block
    - ReadWriteMany
        storage: 32Gi
      name: debian-vm-image
      namespace: vm-library
datavolume spec

This creates a new PVC that will be attached. As we're running a recent version of kubevirt the virtlauncher pod runs as qemu user (= no root permissions).

The issue now is that the blockdevice is owned by root and gid 6 which result in the fact that qemu cannot attach the blockdevice to the VM.

Why blockdev We're using block volumes because this leverages benefits of ceph and eases many processes around the lifecycle.

As a result, the issue shows as following in the logs

{"component":"virt-launcher","kind":"","level":"error","msg":"Failed to sync vmi","name":"debian-vm-clone1","namespace":"vms","pos":"server.go:202","reason":"virError(Code=1, Domain=10, Message='internal error: process exited w
hile connecting to monitor: 2024-04-10T10:08:22.675233Z qemu-kvm: -blockdev {\"driver\":\"host_device\",\"filename\":\"/dev/os-disk\",\"node-name\":\"libvirt-1-storage\",\"cache\":{\"direct\":false,\"no-flush\":false},\"auto-read-only\
":true,\"discard\":\"unmap\"}: Could not open '/dev/os-disk': Permission denied')","timestamp":"2024-04-10T10:08:22.912726Z","uid":"c70ff394-aee1-438e-8eaf-f18bc27a7753"}
error message in virtlauncher

After all - the permission mismatch results in the error that kubevirt can't map the blockdevice into the VM.

The solution is to adjust the containerd settings in such a way that block devices are mounted using the securityContext of the pod as owner.

    device_ownership_from_security_context = true
adjust ownership of block devices / containerd config.yoml

or here

device_ownership_from_security_context = true
adjustment for CRIO runtime

(sources: &

Just apply the change, restart containerd and new pods will be able to attach the disks as expected.


Daniel Nachtrub

Kind of likes computers. Linux foundation certified: LFCS / CKA / CKAD / CKS. Microsoft certified: Cybersecurity Architect Expert & Azure Solutions Architect Expert.