Running kubernetes clusters with isolated workloads (that may be managed by multiple parties) require the use of namespaces to enforce proper isolation of environments.

In the case which lead to this post, we're having such a setup that ingress had been seperated from the applications (which is probably the most common way of deployment).

ingress-namespace
└─ nginx-ingress-controller
└─ default-tls-certificate

application-namespace
└─ services
└─ deployments
└─ ingresses

Now we did not want that the application team needs to manage the TLS secret(s) as there have been some special requirements regarding this. Therefore we placed the default tls certificate into the ingress-namespace and configured the nginx ingress controller deployment with the following settings:

- /nginx-ingress-controller
[...]
- --watch-namespace=application-namespace
- --default-ssl-certificate=ingress-namespace/default-tls-certificate

This looks quite plausible so far. And sort of it is, but let's be honest - if it would, why should i write this post.

Starting the controller shows the following message in the log:

7 controller.go:1102] 
Error loading custom default certificate, falling back to generated default: 
local SSL certificate ingress-namespace/default-tls-certificate was not found

Believe me, I've read this over and over again, I tried adjusting the secret name, checked bindings of the serviceAccount and also tried reading the secret manually using curl from within the controller using the sa token.

TL;DR - How to fix this

Fixing this requires you to do one of both things:

• Move the secret into the application namespace (which is watched)
• Remove watch-namespace
• Use watch-namespace-selector

Move secret into application namespace

Starting nginx controller with watch-namespace enabled, limits the controller at it's core to read only resources that are part of the watched namespace (not sure how about the nginx controller configMap). The layout looks afterwards like this:

ingress-namespace
└─ nginx-ingress-controller

application-namespace
└─ default-tls-certificate
└─ services
└─ deployments
└─ ingresses

This has the clear downside that the application team now can retrieve the contents of the secret (unless they are not allowed to manage or map their own resources within the namespace as there are so many ways to get to the contents).

But - keep in mind:

Remove watch-namespace

The second option is to remove the limitation of watch namespace. This means that every namespace can define ingress resources that will be picked up by this ingress.

Depending on your setup this might be fine. In our case we're having multiple distinct ingresses that are limited to the namespaces, so we cannot remove it.

Use watch-namespace-selector

Besides these very simple options you can also add a label to namespaces that should be watched. In our case that would be the namespace of the application and the namespace of the ingress.

Why?

• Application namespace is managed by the app team
• Ingress namespace is managed by the infra team

Having the namespace selector in place we can reach both goals: Use the ingress namespace to store the certificate secret and use the application namespace for the ingress resources. As long as the infra team does not create ingress resources in the ingress namespace, the controller can be considered exclusive to the application.

Why does this occur?

During analysis i've checked the codebase of the ingress controller to see how the secret is retrieved. Generally it looks like the controller is using some generic store base class that is used to access the configuration backend (kubernetes/etcd in this case).

Assigning a watch-namespace seemingly configures this store with some sort of base filter that limits every incoming request to this namespace(s). Assigning a value (even with namespace prefixed) can (as a result) then not be located by the store.

From a development perspective of the store this is quite nice as it enforces a limitation of the store's scope when accessing data. In this very particular case the limitation might not be intended as the default-tls-certificate is passed as a startup value and might therefore be considered outside of the limitation (as if you would like could then also just drop the watch-namespace argument).

I've added a github issue for this, let's see what will happen.

Using default-tls-certificate & watch-namespace does not work if secret is not in watched namespace · Issue #9196 · kubernetes/ingress-nginx

What happened: Using the combination of watch-namespace and default-tls-certificate flag leads to an error loading the default-tls-certificate if the certificate is not in the namespace that is wat...

GitHubkubernetes