Gateway API: New stuff breaks old things

We recently migrated our GitLab instance to use the Kubernetes Gateway API. This was a somewhat annoying job because our operator-based GitLab deployment installs and manages several dependencies itself, including Cert-Manager, the Gateway API CRDs, and related resources. Using a platform-owned Gateway and GatewayClass, backed by Envoy Gateway, therefore required quite a bit of additional configuration.

Afterwards, the migration appeared to be successful. However, a few days later, we noticed that some pipelines were failing:

$ if command -v glab &> /dev/null; then # collapsed multi-line command
• Creating or updating release repo=devops/helm/keycloak-operator tag=0.3.2

   ERROR

  404 Not Found.

After some debugging, we found that the problem was related to Envoy’s path-normalization behavior.

The GitLab CLI can address projects by their numeric ID or by their URL-encoded project path. In this case, it used the project path, resulting in an API request similar to:

/api/v4/projects/devops%2Fhelm%2Fkeycloak-operator/...

Envoy Gateway received this request, unescaped the encoded slash, and returned a redirect. The redirected path looked something like this:

/api/v4/projects/devops/helm/keycloak-operator/...

That path no longer matched GitLab’s API route for a project identifier, which ultimately resulted in a 404 Not Found response.

This behavior is documented in the Envoy Gateway documentation. It exists for a good reason: normalizing escaped slashes helps protect against path-confusion vulnerabilities, in which different components of a request-processing chain interpret encoded slashes differently. Envoy Gateway’s default action is UnescapeAndRedirect.

For this GitLab deployment, the solution was to configure the Envoy Gateway to preserve escaped slashes. This can be done using a ClientTrafficPolicy that targets the Gateway used by GitLab:

apiVersion: gateway.envoyproxy.io/v1alpha1
kind: ClientTrafficPolicy
metadata:
  name: keep-escaped-slashes
  namespace: gateway
spec:
  targetRefs:
    - group: gateway.networking.k8s.io
      kind: Gateway
      name: envoy-gateway
  path:
    escapedSlashesAction: KeepUnchanged

KeepUnchanged preserves encoded slash sequences such as %2F, allowing GitLab API requests that use URL-encoded project paths to reach GitLab unchanged. GitLab’s Gateway API documentation recommends this setting for Envoy Gateway deployments because GitLab APIs use paths such as /projects/group%2Fproject.

The policy is namespace-scoped, so it must be created in the namespace where the target Gateway resides - in this case, gateway.