Can't delete kubernetes namespace (NamespaceDeletionDiscoveryFailure)

Daniel Nachtrub -

Sometimes - deleting stuff is not as straight forward as you might expect it. In this case, on one of our lab clusters I just wanted to clean some old resources - while i discovered that i'm not able to delete namespaces.

The error shown there has been:

NamespaceDeletionDiscoveryFailure
Discovery failed for some groups, 1 failing: unable to retrieve the complete list of server APIs: packages.operators
.coreos.com/v1: stale GroupVersion discovery: packages.operators.coreos.com/v1

So it seems that some of the tests had some remains and hasn't been cleaned up entirely before.

First approach was of course to

  • delete no longer used CRDs
  • try to remove the finalizer from the namespace

Still - no progress.

Remove old apiservices

As it showed we had from an coreos / operator test a deeply integrated component, an apiservice (which can be seen as a service that extends the API server, see here: https://kubernetes.io/docs/tasks/extend-kubernetes/setup-extension-api-server/). The service & pods had been removed but the definition had been still there.

NAME                                        SERVICE                      AVAILABLE                 AGE
v1.                                         Local                        True                      2y358d
# ... 
v1.mongodbcommunity.mongodb.com             Local                        True                      79d
v1.networking.k8s.io                        Local                        True                      2y358d
v1.node.k8s.io                              Local                        True                      2y358d
v1.packages.operators.coreos.com            olm/packageserver-service    False (ServiceNotFound)   87d
v1.pkg.crossplane.io                        Local                        True                      29d
# ...
v1beta1.metallb.io                          Local                        True                      120m
v1beta1.metrics.k8s.io                      kube-system/metrics-server   True                      2y9d
v1beta1.pkg.crossplane.io                   Local                        True                      29d
# ...

api services

You can quickly spot that the packageserver-service is unavailable.

So, to recover from the situation:

  • Delete the orphaned apiservice (or fix it if it's still required / in use)
  • Wait / issue the delete command again

Easy fix - if you know where to search :-)