Structured YAML diffs for Helm

The problem: diffs that lie about where the change is

Our pipeline-driven Helm deployment workflow runs every chart through --check before it ever touches a cluster. Under the hood, check mode does helm template | kubectl diff --server-side, and until now the actual "what changed" text came from colordiff -u wrapped around kubectl's LIVE and MERGED temp directories. That's a perfectly normal unified diff — and unified diffs are exactly the wrong shape for Kubernetes manifests.

Here's a real example that kept coming up in review: a Rook CephFilesystem CRD losing a few empty sub-keys during server-side apply normalization.

@@ -7,7 +7,7 @@
   finalizers:
   - cephfilesystem.ceph.rook.io
-  generation: 2
+  generation: 3
@@ -41,20 +41,11 @@
     dataPools:
     - application: ""
       erasureCoded:
-        codingChunks: 0
-        dataChunks: 0
       failureDomain: host
-      mirroring: {}
       parameters:
         compression_mode: aggressive
-      quotas: {}

Two unrelated problems, side by side:

1. The hunk header is a lie. @@ -41,20 +41,11 @@ tells you nothing about which resource, which Kubernetes kind, or which YAML path is involved. You have to scroll up through the log — or open the file — to even know you're looking at a CephFilesystem named ceph-filesystem in rook-ceph.
2. Bookkeeping noise drowns real changes. metadata.generation bumps on almost every apply. It shows up as a "change" in every diff, right next to changes operators actually care about, training people to skim past red lines instead of reading them.

That second problem got worse once we added an AI review stage after check mode: the review model reads /tmp/helm_diff_*.txt artifacts under a token budget (pipeline_review_max_tokens). Feeding it unified-diff hunks with misleading headers and generation-number churn meant burning tokens on noise and giving the model less signal to work with per token — the opposite of what you want from an automated reviewer.

We wanted a diff that:

• always tells you which resource changed before showing what changed
• shows changes as a YAML tree with -/+ inline at the right indentation, not generic hunks
• lets us strip known-noisy paths (metadata.generation, metadata.resourceVersion, …) out of the comparison entirely
• collapses everything that didn't change, instead of printing 200 lines of ConfigMap.data to show a 1-line edit
• defaults to zero surrounding context, because parent-key breadcrumbs already tell you where you are
• is precise and compact enough to hand directly to the AI review pipeline as its primary input

The solution: replace the diff display, not the diff engine

The tempting-but-wrong move here is to reimplement kubectl diff's comparison logic yourself. We didn't do that. kubectl diff already does the hard part correctly: it fetches live cluster state, computes the server-side-apply dry-run merge, writes both to temp directories, and then shells out to whatever KUBECTL_EXTERNAL_DIFF points at with those two directories as arguments. That external-diff hook is a stable, documented contract:

KUBECTL_EXTERNAL_DIFF="program [args...]"
program <from_dir> <to_dir>
exit 0   → no differences
exit 1   → differences found
exit >1  → error

So we kept kubectl diff doing exactly what it's good at — SSA-accurate LIVE vs. MERGED comparison — and replaced only the rendering layer behind that hook with a small, dependency-free Python formatter: structured_yaml_diff.py, shipped as an Ansible role library script (same pattern as the existing print_diff.py) and installed on PATH in the container image as kubectl-structured-yaml-diff.

Aside: kubectl 1.34+ parses KUBECTL_EXTERNAL_DIFF as a whitespace-split command where only the first token is treated as the executable — arguments containing / get silently dropped. python3 /path/to/script.py therefore actually runs bare python3 against the diff directories. That's why the formatter is installed as a single-token binary rather than invoked via a path.

How it actually diffs things

Instead of turning two files into strings and diffing lines, the formatter parses each resource with yaml.safe_load into a dict tree and walks both trees together:

1. Load — kubectl names files predictably (apps.v1.Deployment.default.myapp), so files are paired by name across the LIVE/MERGED directories. A resource only in MERGED is "added"; only in LIVE is "removed".
2. Prune excludes — configured dot-notation paths (default: metadata.generation, metadata.resourceVersion) are deep-deleted from both trees before anything is compared. If a resource is identical after pruning, it's reported as unchanged — not "changed, but you should ignore this line."
3. Structural diff — a recursive tree walk tags every node added, removed, modified, or unchanged, matching list items by their name key when present (the Kubernetes convention for things like containers) and falling back to index otherwise.
4. Render — depth-first, starting from the always-present identity header, then walking down: unchanged subtrees with no changed descendants are skipped entirely (collapse); subtrees that do lead to a change print their key as an unmarked breadcrumb and recurse; leaves print as -/+ lines at their real YAML indentation.

That last point is what fixes the CephFilesystem example. The old diff showed a keyless hunk. The new one shows this:

---
apiVersion: ceph.rook.io/v1
kind: CephFilesystem
metadata:
  name: ceph-filesystem
  namespace: rook-ceph
spec:
  dataPools:
-  - application: ""
-    erasureCoded:
-      codingChunks: 0
-      dataChunks: 0
-    failureDomain: host
-    mirroring: {}
+  - failureDomain: host
     parameters:
       compression_mode: aggressive
       target_size_ratio: "0.02"
     replicated:
       size: 3
-    quotas: {}
-    statusCheck:
-      mirror: {}

Notice parameters and replicated appear unprefixed even though they didn't change — not because of a context setting, but because they're siblings inside the changed list item, on the path between spec and the actual edits. Meanwhile the generation bump from the original example doesn't appear at all, because it's in the default exclude list.

Identity headers, as the mirror image of excludes

The single biggest usability win is the identity header — every changed resource block always opens with apiVersion, kind, metadata.name, and metadata.namespace (when present), before any change markers. No more guessing which object a hunk belongs to.

The elegant part is how it's implemented: rather than hardcoding those four fields, we reused the same dot-notation path mechanism as excludes, just pointed in the opposite direction — helm_diff_include_paths. Excludes prune paths out of the comparison; includes force paths into the header unconditionally, whether or not they changed. Same config shape, same code path for path resolution, opposite intent. Want a label always visible in the header for triage purposes? Add it to helm_diff_include_paths — no code change needed.

helm_diff_exclude_paths:
  - metadata.generation
  - metadata.resourceVersion
helm_diff_include_paths:
  - apiVersion
  - kind
  - metadata.name
  - metadata.namespace
helm_diff_context: 0
helm_diff_color: true

One deliberate asymmetry: a missing exclude path is a no-op (deleting something that isn't there does nothing), but a missing include path has to be handled explicitly, because plenty of real Kubernetes objects don't have every field. Cluster-scoped kinds — ClusterRole, ClusterRoleBinding, most CRDs — have no metadata.namespace. If the renderer treated that as an error or printed namespace: null, every cluster-scoped resource diff would be noisy by default. So absence of a configured include path is a silent, expected skip for that resource:

---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: myapp-reader
rules:
-  - resources: ["pods"]
+  - resources: ["pods", "services"]

No blank namespace: line, no warning — just the fields that actually exist.

Zero context by default, full-line color always

Traditional diff tools pad every change with a few lines of surrounding context (-U 3) so you don't lose your place. We default that to 0. In this format, the parent-key breadcrumbs are the context — you already know you're looking at spec.dataPools[0] because the tree walk told you on the way down. Operators who want traditional padding can set helm_diff_context: N and get up to N unchanged sibling lines around each change cluster (context never re-expands a collapsed subtree, so it can't accidentally undo the noise reduction).

For CI logs, we kept full-line ANSI coloring — an entire - line is red, an entire + line is green, not just the marker character — flowing through the same print_diff module that already knew how to preserve escape codes past Ansible's output-stripping. That was a deliberate, separate axis from the AI pipeline: humans get color in the terminal; the AI artifacts get plain text, because ANSI escape sequences are pure token waste for a model.

Feeding this straight into AI review

This is where the two halves of the feature connect. The AI review stage was already reading /tmp/helm_diff_*.txt under a token budget — it just used to receive the same noisy colordiff output as the terminal, ANSI codes included. Now, by default (helm_diff_ai_review: true):

• Artifacts are written as the structured format with ANSI stripped — same tree-shaped, collapsed, identity-headed diff, just plain text.
• review.yml's artifact discovery drops the legacy kubectl_diff_*.txt files for charts that already have a structured helm_diff_*.txt artifact, so the model isn't shown the same change twice in two formats.
• The review prompt template was updated to explain the format explicitly, so the model knows an unmarked line is a breadcrumb, not "no change to report."

The net effect: for a representative large CRD change like the CephFilesystem example, the structured artifact is well over 50% smaller than the equivalent unified diff, while containing more usable signal (the identity header alone answers "what resource is this?", which the model previously had to infer or guess at). Operators who aren't ready to switch can flip helm_diff_ai_review: false and get the pre-feature artifact behavior back, no other changes required.

What this looked like end to end

Before (per resource, terminal and AI input identical, generation noise included):

@@ -7,7 +7,7 @@
-  generation: 2
+  generation: 3

After, for humans (full-line color in CI logs) and for the AI reviewer (same content, ANSI stripped):

---
apiVersion: ceph.rook.io/v1
kind: CephFilesystem
metadata:
  name: ceph-filesystem
  namespace: rook-ceph
spec:
  dataPools:
-  - application: ""
-    erasureCoded:
-      codingChunks: 0
-      dataChunks: 0
+  - failureDomain: host
     parameters:
       compression_mode: aggressive

Same underlying kubectl diff --server-side computation. Same pipeline, same artifact paths, same exit-code contract (0 clean / 2 changed / non-zero on error, via the existing shell remap). The only thing that changed is what gets printed — and that turned out to be enough to fix both the human review experience and the AI review pipeline's input quality at the same time.

Built in under 2 hours, spec first

Worth calling out on its own: this entire feature — spec, plan, research, data model, contracts, implementation, and unit tests — went from a one-paragraph problem statement to done in under two hours.

That's not a "we rushed it" claim, it's the opposite. Spec-driven development front-loaded the ambiguous decisions — what counts as an identity field, how includes and excludes should mirror each other, what happens to a cluster-scoped resource with no metadata.namespace, how AI artifacts and human CI output diverge — into a written spec and plan before any Python was written. By the time implementation started, there was no back-and-forth about intent: the contract for structured_yaml_diff.py, its exit codes, its environment variables, and its exact output format for edge cases were already pinned down on paper. Writing the formatter and its test suite against an unambiguous contract is mechanical work; it's the design churn that usually eats the hours, and the spec absorbed that churn up front instead of during implementation or code review.