Change statefulset spec without downtime
Altering statefulsets on kubernetes can be tricky - as statefulsets are very common used for persistent applications like databases recreation is no option. This guide shows a path around some of these limitations.
The most common type of application deployments in kubernetes are Deployments and DaemonSets. Deployments are mostly used for any stateless services and DaemonSets for host-bound helpers, for example monitoring and metrics gathering pods.
Besides those, StafulSets are quite useful - obviously on all kinds of applications that require state (more on this) and/or for example deterministics names. Yes - it is possible to attach volumes to deployments, so these can also be deterministics. But this makes only sense if you use volumes that can be attached to multiple instances at the same time (ReadWriteMany, configMaps, secrets, and so on). If you need to work with RWO volumes it's not possible on Deployments to mount seperate volumes per pod.
Why is then StatefulSet not used more often?
The thing with StatefulSets is that altering them can be tricky because they are very rigid and updating them is limited to only a few properties. Therefore you might have seen the following message:
spec: Forbidden: updates to statefulset spec for fields other than ‘replicas’, ‘template’, and ‘updateStrategy’ are forbiddenThis makes the StatefulSet hard to manage on an ever evolving environment.
Changing a statefulset spec without downtime
Luckily, there are ways to bypass such limitations - if you understand what you are doing and what limitations apply.
The easiest way is to do the following procedure:
- Export the statefulset spec as yaml
- Alter the spec according to your requirements
Tip: Do not modify the selector, this might cause issues - Delete statefulset with cascade=orphan (pods will not be deleted)
- Apply the modified spec (which will recreate the sts)
Having done these steps, the configuration has been updated. Running pods will be associated again with the statefulset and everything is fine.
This path is a bypass around an intended limitation of kubernetes - so let's consider this we are slightly bending the intended use :-)
It is clear that you can't expect to be able to alter anything on the statefulset spec and it works. A usecase for me is for example to adjust the volumeClaimTemplates (most likely an increase in size) in combination with an update of the volumes on another path.
