microk8s - change location of hostpath storage

From time to time we need small container hosts - very small, single hosts. To work with these and still get benefits of gitops deployments, we use microk8s which has reasonable footprint.

On our installations we're strictly moving persistent data to dedicated disks and do not store these on the operating system disks (for various reasons). So, our layout looks for like this:

NAME   MAJ:MIN RM   SIZE RO TYPE MOUNTPOINT
sda      8:0    0    64G  0 disk 
├─sda1   8:1    0   512M  0 part /boot/efi
└─sda2   8:2    0  63.5G  0 part /
sdb      8:16   0   128G  0 disk 
└─sdb1   8:17   0   128G  0 part /mnt/data

disk layout

As these hosts are by design not meant to be scaled out, we want to avoid using a replicated storage like ceph or mayastor as this adds overhead in terms of memory and reduces throughput. We want a very simple system, hostPath storage.

Long story short: We enable hostpath storage according to the documentation with microk8s enable hostpath-storage.

But - then all PVs will default to /var/snap/microk8s/common/default-storage/ and we don't want this. We want the location be /mnt/data/microk8s.

To adjust this, the documentation states to create a new storageClass - which didn't work in our case. We needed to adjust the deployment (namespace: kube-system) of the hostpath-provisioner as the hostPath root for the provisioning is mapped there. Additional (to make things more clear) we provided a custom, default storageClass without a pvDir parameter.

apiVersion: apps/v1
kind: Deployment
metadata:
  annotations:
  labels:
    k8s-app: hostpath-provisioner
  name: hostpath-provisioner
  namespace: kube-system
spec:
  progressDeadlineSeconds: 600
  replicas: 1
  revisionHistoryLimit: 0
  selector:
    matchLabels:
      k8s-app: hostpath-provisioner
  strategy:
    rollingUpdate:
      maxSurge: 25%
      maxUnavailable: 25%
    type: RollingUpdate
  template:
    metadata:
      creationTimestamp: null
      labels:
        k8s-app: hostpath-provisioner
    spec:
      containers:
      - env:
        - name: NAMESPACE
          valueFrom:
            fieldRef:
              apiVersion: v1
              fieldPath: metadata.namespace
        - name: NODE_NAME
          valueFrom:
            fieldRef:
              apiVersion: v1
              fieldPath: spec.nodeName
        - name: PV_DIR
          value: /mnt/data/microk8s
        - name: BUSYBOX_IMAGE
          value: busybox:1.28.4
        image: cdkbot/hostpath-provisioner:1.2.0
        imagePullPolicy: IfNotPresent
        name: hostpath-provisioner
        resources: {}
        terminationMessagePath: /dev/termination-log
        terminationMessagePolicy: File
        volumeMounts:
        - mountPath: /var/snap/microk8s/common/default-storage
          name: pv-volume
      dnsPolicy: ClusterFirst
      restartPolicy: Always
      schedulerName: default-scheduler
      securityContext: {}
      serviceAccount: microk8s-hostpath
      serviceAccountName: microk8s-hostpath
      terminationGracePeriodSeconds: 30
      volumes:
      - hostPath:
          path: /mnt/data/microk8s
          type: ""
        name: pv-volume
---
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  annotations:
    storageclass.kubernetes.io/is-default-class: "true"
  name: hostpath-data
provisioner: microk8s.io/hostpath
reclaimPolicy: Retain
volumeBindingMode: WaitForFirstConsumer
allowVolumeExpansion: true

adjusted provisioner

As you can see, the only thing we adjusted is the mountPath and PV_DIR within the deployment.

Newly created PVs will now be placed on the new location - would be nice if the manual would be updated to reflect this requirement. Enjoy!