Can't delete kubernetes namespace (NamespaceDeletionDiscoveryFailure)
While trying to delete a namespace on a kubernetes cluster I learned that there is a resource type apiservice :-) This post shows how to unblock deletions due orphaned apiservice definitions.
Sometimes - deleting stuff is not as straight forward as you might expect it. In this case, on one of our lab clusters I just wanted to clean some old resources - while i discovered that i'm not able to delete namespaces.
The error shown there has been:
NamespaceDeletionDiscoveryFailure
Discovery failed for some groups, 1 failing: unable to retrieve the complete list of server APIs: packages.operators
.coreos.com/v1: stale GroupVersion discovery: packages.operators.coreos.com/v1
So it seems that some of the tests had some remains and hasn't been cleaned up entirely before.
First approach was of course to
- delete no longer used CRDs
- try to remove the finalizer from the namespace
Still - no progress.
Remove old apiservices
As it showed we had from an coreos / operator test a deeply integrated component, an apiservice (which can be seen as a service that extends the API server, see here: https://kubernetes.io/docs/tasks/extend-kubernetes/setup-extension-api-server/). The service & pods had been removed but the definition had been still there.
NAME SERVICE AVAILABLE AGE
v1. Local True 2y358d
# ...
v1.mongodbcommunity.mongodb.com Local True 79d
v1.networking.k8s.io Local True 2y358d
v1.node.k8s.io Local True 2y358d
v1.packages.operators.coreos.com olm/packageserver-service False (ServiceNotFound) 87d
v1.pkg.crossplane.io Local True 29d
# ...
v1beta1.metallb.io Local True 120m
v1beta1.metrics.k8s.io kube-system/metrics-server True 2y9d
v1beta1.pkg.crossplane.io Local True 29d
# ...api services
You can quickly spot that the packageserver-service is unavailable.
So, to recover from the situation:
- Delete the orphaned apiservice (or fix it if it's still required / in use)
- Wait / issue the delete command again
Easy fix - if you know where to search :-)
