You've successfully subscribed to Nuvotex Blog
Great! Next, complete checkout for full access to Nuvotex Blog
Welcome back! You've successfully signed in.
Success! Your account is fully activated, you now have access to all content.
Success! Your billing info is updated.
Billing info update failed.

Manually remove a member from etcd

Daniel Nachtrub
Daniel Nachtrub

Recently, I needed to rollback a kubernetes control-plane node to an older snapshot. This cause (obviously) that etcd could not operate anymore in the etcd cluster.

The approach in this scenario is to remove the etcd node from the cluster and add it again. The removal of it is rather easy.

  • Take one of the remaining nodes and exec into the etcd container
  • List all members
  • Drop the affected member

To list all members and remove the desired one, just two commands are required.

# list members
etcdctl member list --endpoints= --cacert=/etc/kubernetes/pki/etcd/ca.crt --cert=/etc/kubernetes/pki/etcd/peer.crt --key=/etc/kubernetes/pki/etcd/peer.key

# remove
etcdctl member remove <MEMBERID> --endpoints= --cacert=/etc/kubernetes/pki/etcd/ca.crt --cert=/etc/kubernetes/pki/etcd/peer.crt --key=/etc/kubernetes/pki/etcd/peer.key
list and remove member

That's it - now you can join back the member to the cluster and it will replicate all data.

Daniel Nachtrub

Kind of likes computers. Linux foundation certified: LFCS / CKA / CKAD / CKS. Microsoft certified: Cybersecurity Architect Expert & Azure Solutions Architect Expert.