SilverBack

WiP

Does the heavy lifting.

The one with the back patch.

Backpatch

Backpatch reconciliation workflow

The aim of this is to enable GitOps+Kubernetes' declarative paradigm without resorting to constant git commits. You "wire up" your resources in the Kubernetes control plane and let the runtime reconcilliation ensure the values are current.

Current state:

• Source configMap objects may be specified as inputs.
• Source configMaps are labeled as sources, watched for changes, and unlabelled when no longer in use.
• Source configMap marker labels are managed using server-side apply.
• Destination resources are arbitrary and nested.
• Destination resources are owned and automatically handled by native finalizers.
• Object patches are applied via merge in-memory.

Issues and discussion

Multi-sourced ConfigMaps rely on a delimiter that has to be part of the allowed character set for label values. This means that Backpatch names could potentially foul delimiters. Change marker label delimiter to something unlikely to accidentally occur, and restrict Backpatch names from containing the delimiter. The latter is a bit tail-wag-dog though.

There's no checks on multi-sourced configMap marker labels for length. It shouldn't be too bad to switch to one well-known marker label for watch predicates, and then a templated label value for each Backpatch downstream. This still smells a bit of label abuse and blowout.

Forcing configMap resources for data export is verbose. This is a trade-off, it's certainly more verbose but that's also explicit. Allowing arbitrary resource sources fixes this but... This trades off a lot of simplicity in sources as now a full GVK will be required for all sources. We could default to configMaps, or make there be two source input lists. I'll put this last as it's not only not strictly necessary but watching arbitrary input resources for reconciliation is going to be a fight. This also means allowing arbitrary source paths. Now sjson I think builds on gjson, which may mean the path syntax is common for retrieval as well.

Testing doesn't work. This is due to deep Kubernetes limitations with unstructured. I may be able to hack around it because the test data is well-known, so I can use proper structs/types. Or I can use the full naked Kube client, bypassing the cachey abstraction that causes this?

Patching paths are off-standard for Kubernetes. Kubernetes uses JsonPath, which most notably uses [0] syntax for array indexing. Our patching library, sjson, uses .0 syntax. I spent a while trying to get the official k8s GoLang library for JsonPath to work for patching. JsonPath and the library are read-only though.

Nested destination resources. This sacrifices a lot of tooling and convenience for users. You no longer get autocompletes, or the everything-on-a-page view of declarative. Allow patching of existing arbitrary resources not created/owned by the controller. Not owning resources means we'll need a full cleanup/management logic. If we remove the creation ability, then users are locked into creating resources they know are either invalid or irreconcilable until patched. In the case of invalid it simply won't apply, irreconcilable is more tolerable but could cause rate limiting issues due to bad reconciliation requests. ACK, at least however, has the concept of a terminal state, where reconciliation is ceased and the resource must be entirely recreated for anything to happen. We could allow an option to create-if-not-found but that's just awful for debugging and race conditions.

Created resources don't display in ArgoCD. I haven't looked into this but potentially it's just a case of propagating annotations and labels. Annotations we're already propagating, labels could be added.

(削除) configMap values can only be strings, which doesn't work for say, port numbers. (削除ここまで) Source values are now interpreted according to YAML. It's a very dumb implementation, also round-tripping strings, but it works. In niche cases, we may need to escape stuff inside the configMap string value to keep it a string.

(削除) A source configMap can't be referenced by more than one backPatch. (削除ここまで) Marker label now supports multiples using a delimiter

(削除) Limited use of Server-side Apply. This means we're probably doing more logic work and field ownership than need be. This is especially relevant to working with objects under multiple controllers. (削除ここまで) This is fixed for source configMap label management. TBD if we implement unowned destination resources.

(削除) JsonPath implementation does not support array indexing. I have immediately one use case that would require this. It looks like both GoLang slices and YAML lists are ordered, though this still feels fraught. Is a list field manager set per list item or the whole thing? Appending or prepending any list items changes the order for all and there's no way to locate or re-locate what you patched. We can't label or annotate the list item either to mark it, and you can't search the list for known values if anything mutated the list entry you were operating on. (削除ここまで) I wasn't able to get util/jsonpath to work for patching. It's fixed but by round-tripping to raw JSON, oh well.

Enhancements

• Allow secret sources
• Convert all client mutation operations to SSA. This is critical for ArgoCD as we'll need to lean on a system-wide ignore of the managed fields manager. This is mostly done, the status/condition updates to the backPatches as well as the resource creation are outstanding.
• Get tests running. Testing comprehensively-enough for it to be meaningful is actually quite an undertaking. It is crucial for any collaborators though and will certainly speed my iteration. This can happen immediately after the minimum utility is built.
• Remove round-trips to strings when evaluating YAML values and patching nested values.
• Use standard JsonPath for patching paths.
• Update the status as we go
• Provide Ready condition
• Check cond.observedGeneration vs metadata.generation
• Check all patch configMapSource are actually specified in the configMapSources.

Notes and References

• Kubebuilder unstructured type issue

• Operator SDK unstructured type issue

• Another option to reverse out the type code

• Runtime.Object search

• Converting runtime.Object

• Watching external resources

• Runtime/unstructured watches

• Runtime/unstructured lists

• Excellent blog on controllers

Dynamic objects

• Dynamic informers
• https://stackoverflow.com/questions/53341727/how-to-submit-generic-runtime-object-to-kubernetes-api-using-client-go
• https://stackoverflow.com/questions/61200605/generic-client-get-for-custom-kubernetes-go-operator?rq=3
• https://stackoverflow.com/questions/65074930/how-to-perform-crud-on-3rd-party-custom-resource-for-which-go-api-is-not-availab?noredirect=1&lq=1
• https://simplerize.com/kubernetes/apply-yaml-manifests-to-kubernetes-using-go-client
• https://stackoverflow.com/questions/58783939/using-client-go-to-kubectl-apply-against-the-kubernetes-api-directly-with-mult
• Dynamic watches issue
• Example cm source code
• SSA example

Testing TypeMeta bug

• https://github.com/kubernetes-sigs/controller-runtime/issues/1517
• https://github.com/kubernetes/kubernetes/issues/80609