You've successfully subscribed to Nuvotex Blog
Great! Next, complete checkout for full access to Nuvotex Blog
Welcome back! You've successfully signed in.
Success! Your account is fully activated, you now have access to all content.
Success! Your billing info is updated.
Billing info update failed.

Change statefulset spec without downtime

Altering statefulsets on kubernetes can be tricky - as statefulsets are very common used for persistent applications like databases recreation is no option. This guide shows a path around some of these limitations.

Daniel Nachtrub
Daniel Nachtrub

The most common type of application deployments in kubernetes are Deployments and DaemonSets. Deployments are mostly used for any stateless services and DaemonSets for host-bound helpers, for example monitoring and metrics gathering pods.

Besides those, StafulSets are quite useful - obviously on all kinds of applications that require state (more on this) and/or for example deterministics names. Yes - it is possible to attach volumes to deployments, so these can also be deterministics. But this makes only sense if you use volumes that can be attached to multiple instances at the same time (ReadWriteMany, configMaps, secrets, and so on). If you need to work with RWO volumes it's not possible on Deployments to mount seperate volumes per pod.

Why is then StatefulSet not used more often?

The thing with StatefulSets is that altering them can be tricky because they are very rigid and updating them is limited to only a few properties. Therefore you might have seen the following message:

spec: Forbidden: updates to statefulset spec for fields other than ‘replicas’, ‘template’, and ‘updateStrategy’ are forbidden
updating a statefulset is limited

This makes the StatefulSet hard to manage on an ever evolving environment.

Changing a statefulset spec without downtime

Luckily, there are ways to bypass such limitations - if you understand what you are doing and what limitations apply.

The easiest way is to do the following procedure:

  • Export the statefulset spec as yaml
  • Alter the spec according to your requirements
    Tip: Do not modify the selector, this might cause issues
  • Delete statefulset with cascade=orphan (pods will not be deleted)
  • Apply the modified spec (which will recreate the sts)
Keep UIDThe pods have assigned ownerReferences that are linked the UID of your statefulset - make sure to keep this UID when applying the statefulset. Otherwise the reference will be invalid.

Having done these steps, the configuration has been updated. Running pods will be associated again with the statefulset and everything is fine.

This path is a bypass around an intended limitation of kubernetes - so let's consider this we are slightly bending the intended use :-)

It is clear that you can't expect to be able to alter anything on the statefulset spec and it works. A usecase for me is for example to adjust the volumeClaimTemplates (most likely an increase in size) in combination with an update of the volumes on another path.


Daniel Nachtrub

Kind of likes computers. Linux foundation certified: LFCS / CKA / CKAD / CKS. Microsoft certified: Cybersecurity Architect Expert & Azure Solutions Architect Expert.