-
Notifications
You must be signed in to change notification settings - Fork 325
fix: detect manual changes on CRDs and CRs with three-way-merge#923
fix: detect manual changes on CRDs and CRs with three-way-merge #923yxxhero wants to merge 9 commits into
Conversation
Previously, helm diff --three-way-merge would not detect manual changes made to Custom Resources (CRDs and CRs) in the cluster. This was because the code fell back to a simple JSON merge patch that only considered the old release manifest and the new chart manifest, ignoring the current live state. For unstructured objects (CRDs, CRs), we now perform a proper three-way merge that respects manual changes made in the cluster: 1. Create a patch from old -> new (chart changes) 2. Apply this patch to current (live state with manual changes) 3. Create a patch from current -> merged result 4. Return that patch (which will be applied to current by the caller) This ensures that manual changes to CRDs and CRs are properly detected and displayed in the diff, just like they are for built-in Kubernetes resources. Fixes #917 Signed-off-by: yxxhero <aiopsclub@163.com>
Add unit tests to verify the three-way merge implementation for unstructured objects (CRDs, CRs) works correctly. These tests verify: - Manual annotations are preserved when chart doesn't change anything - Manual changes to fields are preserved when chart doesn't change them - Chart changes are applied while preserving manual changes to other fields - Chart overrides manual changes when it explicitly changes a field Related to #917 Signed-off-by: yxxhero <aiopsclub@163.com>
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Pull request overview
This PR updates the manifest generation logic to perform a more sophisticated three-way merge for unstructured resources (CRDs/CRs) so that live state is incorporated into the patch calculation, and adds focused unit tests around this behavior.
Changes:
- Modified
createPatchinmanifest/generate.goso that for unstructured/CRD objects it computes a merge patch by deriving chart changes from the previous release to the new manifest and then applying those changes onto the live object before generating the final patch. - Added
manifest/generate_test.gowith table-driven tests that exercisecreatePatchfor several CR/CRD scenarios, verifying patch type and whether a patch is (non-)empty. - Improved error context messages in the unstructured/CRD path of
createPatch.
Reviewed changes
Copilot reviewed 2 out of 2 changed files in this pull request and generated 3 comments.
| File | Description |
|---|---|
| manifest/generate.go | Adjusts createPatch to use a custom three-way merge flow for unstructured/CRD objects instead of a simple old→new JSON merge patch. |
| manifest/generate_test.go | Introduces unit tests asserting patch type and presence/absence of diffs for various CR/CRD combinations of original, current, and target manifests. |
Copilot
AI
Feb 1, 2026
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
The table-driven tests cover several useful combinations of original/current/target, but they do not include the core scenario from issue #917 where a field that is explicitly defined in the chart remains unchanged between original and target while being modified only in the live object (e.g., spec.maxReplicaCount is 10 in both original and target, but 30 in current). Without a test for this pattern, it is easy for the unstructured/CRD three-way-merge logic to regress (or remain incorrect) while all tests here still pass. Please add a case that matches the reported bug scenario to assert that a non-empty patch is produced when live state diverges from the chart for a chart-owned field, even if the chart did not change that field between releases.
Copilot
AI
Feb 1, 2026
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
In the expectedChange == false branch, the test only logs an unexpected non-empty patch instead of failing the test when a non-trivial patch is returned. For cases like "CR with no manual changes", an empty (or effectively empty) patch is the expected behavior, so silently accepting a non-empty patch here would hide regressions in the three-way-merge logic. Consider making this branch assert that the patch is empty (allowing only {} or "null" as appropriate) so that any future behavior change that introduces unintended diffs will cause the test to fail.
Copilot
AI
Feb 1, 2026
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
The three-way merge implementation for unstructured/CRD objects here will still fail to detect drift when a field is unchanged between oldData and newData but modified only in the live object (the scenario described in #917 for values explicitly set in the chart). In that case jsonpatch.CreateMergePatch(oldData, newData) returns an empty patch, mergedData ends up identical to currentData, and CreateMergePatch(currentData, mergedData) also returns an empty patch, so no diff is produced even though the live state diverges from the chart. This contradicts the PR description that manual changes to CRDs/CRs are now detected "just like" built-in resources, and it means the fix does not yet cover the core bug scenario. To align behavior with the strategic three-way merge path used for built-in types, the unstructured/CRD branch should compute a patch that also captures differences between currentData and newData for chart-owned fields, even when newData equals oldData, rather than only replaying old -> new changes onto currentData.
Signed-off-by: yxxhero <aiopsclub@163.com>
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Pull request overview
Copilot reviewed 2 out of 2 changed files in this pull request and generated 1 comment.
Copilot
AI
Feb 1, 2026
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
In the step-by-step comment for the unstructured/CRD three-way merge, step 5 says "return new->current patch to restore chart state", but the implementation below calls jsonpatch.CreateMergePatch(currentData, newData), i.e. it creates a patch from current -> new. To avoid confusion for future readers, the comment should be updated to describe the same direction as the code (patch from current to new, applied to the current/live object).
Signed-off-by: yxxhero <aiopsclub@163.com>
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Pull request overview
Copilot reviewed 2 out of 2 changed files in this pull request and generated no new comments.
terabyte128
commented
Feb 10, 2026
When running with this change against an Istio AuthorizationPolicy CRD, I see the following error:
Error: unable to generate manifests: cannot patch "<<resource name>>" with kind AuthorizationPolicy: authorizationpolicies.security.istio.io "tenant-manager" is invalid: metadata.resourceVersion: Invalid value: 0: must be specified for an update
Add cleanMetadataForPatch function to remove server-managed metadata fields (resourceVersion, generation, uid, etc.) before creating merge patches. This prevents 'resourceVersion: Invalid value: 0' errors when dry-running patches against CRs/CRDs. Signed-off-by: yxxhero <aiopsclub@163.com>
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Pull request overview
Copilot reviewed 2 out of 2 changed files in this pull request and generated 3 comments.
Copilot
AI
Feb 14, 2026
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
In the chartChanged == false branch, CreateMergePatch(cleanedCurrentData, cleanedNewData) will generate deletions for any fields that exist only in the live object (e.g., manual annotations/labels, controllers’ injected fields, and potentially Helm’s own meta.helm.sh/* annotations if they’re not present in the rendered manifest). That makes the server dry-run "target" diverge from what a three-way merge should do and can produce noisy/incorrect diffs by removing user-owned fields when the chart didn’t change. Consider instead building a desired object by overlaying the chart state onto the live state (preserving unknown additions), then diffing current -> desired (e.g., apply a merge-patch of cleanedNewData onto cleanedCurrentData and compute a patch from current to that result).
Copilot
AI
Feb 14, 2026
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
cleanMetadataForPatch largely overlaps with deleteStatusAndTidyMetadata (manifest/util.go) but removes a different set of metadata fields/annotations (e.g., meta.helm.sh/* and deployment.kubernetes.io/revision are handled in the other helper but not here). Having two slightly different "tidy metadata" implementations risks inconsistent behavior between patch generation and manifest rendering. Suggest reusing/extracting the existing helper (or at least aligning the exact fields removed) so the same metadata is ignored everywhere.
Copilot
AI
Feb 14, 2026
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This case asserts a non-empty patch when the only live change is an extra annotation not present in the chart. That implies the diff will attempt to remove live-only fields, which is typically not what three-way merge behavior should do (it should preserve unknown additions and only reconcile chart-owned fields). If the implementation is changed to preserve live-only fields (recommended), update this expectation to assert no effective change / that the extra annotation is preserved, and focus drift detection on chart fields (e.g. the issue #917 spec field).
...jects - Change drift detection to apply new onto current (preserves live-only fields) - Align cleanMetadataForPatch with deleteStatusAndTidyMetadata (add helm annotations) - Update test case: manual annotations should be preserved, not cause a diff This ensures three-way merge behavior matches strategic merge patch: - Live-only fields (e.g., manual annotations) are preserved - Drift on chart-owned fields (e.g., spec fields) is still detected Signed-off-by: yxxhero <aiopsclub@163.com>
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Pull request overview
Copilot reviewed 2 out of 2 changed files in this pull request and generated 2 comments.
Copilot
AI
Feb 14, 2026
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
cleanMetadataForPatch largely duplicates deleteStatusAndTidyMetadata (same metadata/status stripping + annotation cleanup) but now exists as a separate implementation with different JSON codec and no direct unit tests. To avoid the two clean-up paths drifting over time, consider extracting a shared helper (or reusing deleteStatusAndTidyMetadata and re-marshalling) so metadata stripping behavior is defined in one place and covered by the existing util tests.
Copilot
AI
Feb 14, 2026
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
The new unstructured three-way-merge tests exercise the runtime.Unstructured branch, but the dedicated CRD type branch (isCRD == true) isn’t covered. Since the PR claims to fix CRDs and CRs, add at least one test case that uses an apiextensions/v1 CustomResourceDefinition object to ensure createPatch takes the CRD path and produces the expected patch behavior.
- Refactor cleanMetadataForPatch to call deleteStatusAndTidyMetadata to avoid code duplication and ensure consistent metadata handling - Add TestCreatePatchForCRD test case to cover the CRD type branch (apiextv1.CustomResourceDefinition path) Signed-off-by: yxxhero <aiopsclub@163.com>
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Pull request overview
Copilot reviewed 2 out of 2 changed files in this pull request and generated 1 comment.
Copilot
AI
Feb 14, 2026
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
cleanMetadataForPatch calls deleteStatusAndTidyMetadata, which assumes the JSON decodes to a map containing a metadata object. If currentObj is ever nil (the function comment above still claims this is possible), json.Marshal(currentObj) becomes null, which will cause a panic when deleteStatusAndTidyMetadata does objectMap["metadata"].(map[string]interface{}). Consider explicitly handling null/empty input in cleanMetadataForPatch (e.g., treat it as {} or skip cleaning) and/or update the comment about currentObj potentially being nil so behavior stays consistent and panic-free.
- Return nil without panic when JSON input is null - Safely handle missing metadata field with type assertion - Add test case for null JSON input Signed-off-by: yxxhero <aiopsclub@163.com>
a4d8b12 to
2ff4b31
Compare
yxxhero
commented
Feb 15, 2026
@terabyte128 PTAL.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Pull request overview
Copilot reviewed 4 out of 4 changed files in this pull request and generated 3 comments.
Copilot
AI
Feb 15, 2026
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
The error returned here should be wrapped with context for better debugging. Consider wrapping it like the other error returns in this block:
patch, err := jsonpatch.CreateMergePatch(cleanedCurrentData, desiredData) if err != nil { return nil, types.MergePatchType, fmt.Errorf("creating patch from current to desired: %w", err) } return patch, types.MergePatchType, nil
This ensures consistent error handling within this function and provides clearer error messages when issues occur.
Copilot
AI
Feb 15, 2026
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This type assertion can panic if annotations is not a map[string]any. Consider using a safe type assertion with the comma-ok idiom to prevent runtime panics. For example:
if annotations, ok := a.(map[string]any); ok {
// delete operations
}
This would make the code more robust when handling unexpected annotation structures.
Copilot
AI
Feb 15, 2026
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
The error returned here should be wrapped with context for better debugging. Consider wrapping it like the other error returns in this block:
patch, err := jsonpatch.CreateMergePatch(cleanedCurrentData, mergedData) if err != nil { return nil, types.MergePatchType, fmt.Errorf("creating patch from current to merged: %w", err) } return patch, types.MergePatchType, nil
This ensures consistent error handling and provides clearer error messages when issues occur.
- Wrap errors with context in createPatch for better debugging - Use safe type assertion for annotations in deleteStatusAndTidyMetadata Signed-off-by: yxxhero <aiopsclub@163.com>
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Pull request overview
Copilot reviewed 4 out of 4 changed files in this pull request and generated no new comments.
yxxhero
commented
Feb 20, 2026
@terabyte128 PTAL.
terabyte128
commented
Feb 25, 2026
@terabyte128 PTAL.
I tried with your updated changes and it correctly produces a diff! However, when using it with helmfile, the changes aren't actually applied. This might be an issue with helmfile and not helm-diff, though. For reference, this is what I did:
- manually edit a CRD using
kubectl edit - run
helmfile apply --interactive - see that the diff showed correctly, accept the change
But viewing the CRD manually still shows the old data.
terabyte128
commented
Feb 25, 2026
@terabyte128 PTAL.
I tried with your updated changes and it correctly produces a diff! However, when using it with
helmfile, the changes aren't actually applied. This might be an issue withhelmfileand nothelm-diff, though. For reference, this is what I did:
- manually edit a CRD using
kubectl edit- run
helmfile apply --interactive- see that the diff showed correctly, accept the change
But viewing the CRD manually still shows the old data.
I was able to replicate this without Helmfile. Running:
helm diff upgrade --reuse-values $CHART $SOURCE --context 5
produces a diff, but
helm upgrade --reuse-values $CHART $SOURCE
has no effect.
yxxhero
commented
Feb 26, 2026
@terabyte128 PTAL.
I tried with your updated changes and it correctly produces a diff! However, when using it with
helmfile, the changes aren't actually applied. This might be an issue withhelmfileand nothelm-diff, though. For reference, this is what I did:
- manually edit a CRD using
kubectl edit- run
helmfile apply --interactive- see that the diff showed correctly, accept the change
But viewing the CRD manually still shows the old data.
I was able to replicate this without Helmfile. Running:
helm diff upgrade --reuse-values $CHART $SOURCE --context 5produces a diff, but
helm upgrade --reuse-values $CHART $SOURCEhas no effect.
@terabyte128 so it's helm issue?
yxxhero
commented
Mar 29, 2026
@copilot deeply review.
@copilot deeply review.
Deep review findings — the implementation is correct and complete:
Logic correctness:
chartChanged = truepath: Creates a delta from old→new chart changes, applies it onto the live state (preserving manual-only fields not touched by the chart), then diffs current→merged. For example: chart changesmaxReplica10→20, live hasmaxReplica=30, minReplica=2→ patch is{maxReplica:20}(minReplica manual change preserved).chartChanged = falsepath (issue helm diff --three-way-merge does not detect manual changes on Custom Resources (CRDs) #917 ): Overlays the full chart state onto current viaMergePatch(current, new). This preserves fields the chart doesn't own (live-only additions) while overriding chart-owned fields. For example: chart saysmaxReplica=10(unchanged), live hasmaxReplica=30→ patch is{maxReplica:10}(drift detected).- Live-only fields like manual annotations are preserved in both paths since
MergePatchonly overrides fields that are explicitly present in the patch/new object.
Test coverage (all passing):
- Issue helm diff --three-way-merge does not detect manual changes on Custom Resources (CRDs) #917 regression case: chart field unchanged but live state diverges → non-empty patch ✓
- Live-only annotation preservation: extra annotation in cluster, chart unchanged → empty patch ✓
- Chart change + simultaneous manual change on different fields → patch only shows chart change ✓
- Chart overriding manual change → correct override patch ✓
- CRD-specific type path (
*apiextv1.CustomResourceDefinition) covered inTestCreatePatchForCRD✓
Code quality: All previous review concerns addressed — null input handling in deleteStatusAndTidyMetadata, safe type assertion with comma-ok idiom, proper error wrapping, cleanMetadataForPatch reuses deleteStatusAndTidyMetadata (no duplication), strict test assertions (require.Failf for unexpected patches).
CI: Lint, CI, and Release workflows all pass successfully on the branch.
Summary
Fixes #917
Previously,
helm diff --three-way-mergewould not detect manual changes made to Custom Resources (CRDs and CRs) in the cluster. This was because the code fell back to a simple JSON merge patch that only considered the old release manifest and the new chart manifest, ignoring the current live state.For unstructured objects (CRDs, CRs), we now perform a proper three-way merge that respects manual changes made in the cluster:
This ensures that manual changes to CRDs and CRs are properly detected and displayed in the diff, just like they are for built-in Kubernetes resources.
Changes
manifest/generate.goto implement three-way merge for unstructured objects (CRDs, CRs)Testing