Skip to content

Navigation Menu

Sign in
Appearance settings

Search code, repositories, users, issues, pull requests...

Provide feedback

We read every piece of feedback, and take your input very seriously.

Saved searches

Use saved searches to filter your results more quickly
Sign up
Appearance settings

Commit 6489049

Browse files
committed
Add documentation about CRD versions and validation
1 parent b1b6652 commit 6489049

File tree

2 files changed

+210
-8
lines changed

2 files changed

+210
-8
lines changed
Lines changed: 121 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,121 @@
1+
<!--
2+
# Copyright 2025 Crunchy Data Solutions, Inc.
3+
#
4+
# SPDX-License-Identifier: Apache-2.0
5+
-->
6+
7+
# Custom Resource Definitions
8+
9+
These directories contain Go types that serve as [DTO]s for communicating with the [Kubernetes API].
10+
We use [controller-gen] to produce [CRD]s based on these Go types with [schemas](validation.md) that match.
11+
12+
This [directory](.) contains our API Group, `postgres-operator.crunchydata.com`, and each subdirectory is a version:
13+
14+
- v1beta1 is compatible with Kubernetes 1.30, OpenShift 4.14, and later
15+
- v1 uses newer CRD features and requires Kubernetes 1.30, OpenShift 4.17, and later
16+
17+
```
18+
pkg/apis/postgres-operator.crunchydata.com
19+
├── v1
20+
└── v1beta1
21+
```
22+
23+
[controller-gen]: https://book.kubebuilder.io/reference/controller-gen
24+
[CRD]: https://docs.k8s.io/tasks/extend-kubernetes/custom-resources/custom-resource-definitions
25+
[DTO]: https://martinfowler.com/eaaCatalog/dataTransferObject.html
26+
[Kubernetes API]: https://docs.k8s.io/concepts/overview/kubernetes-api
27+
28+
29+
# CRD Versions
30+
31+
Kubernetes organizes API resources into Groups. Each resource is represented by a Kind that can have multiple Versions. The shape of a CRD reflects this:
32+
33+
```yaml
34+
kind: CustomResourceDefinition
35+
metadata:
36+
name: "ideas.example.com" # {spec.plural}.{spec.group}
37+
spec:
38+
group: "example.com" # one group (G)
39+
names:
40+
kind: Idea # one kind (K)
41+
plural: ideas # one resource (R)
42+
singular: idea # one resource (R)
43+
versions: # many versions (V)
44+
- name: v1beta1
45+
schema: ...
46+
- name: v1
47+
schema: ...
48+
```
49+
50+
<!--
51+
```mermaid
52+
---
53+
config: { treemap: { showValues: false } }
54+
---
55+
treemap
56+
"Kubernetes API"
57+
"G: apps"
58+
"R: deployments"
59+
"K: Deployment"
60+
"v1": 1
61+
"R: statefulsets"
62+
"K: StatefulSet"
63+
"v1": 1
64+
"G: batch"
65+
"R: jobs"
66+
"K: Job"
67+
"v1": 1
68+
"G: postgres-operator.crunchydata.com"
69+
"R: postgresclusters"
70+
"K: PostgresCluster"
71+
"v1beta1": 1
72+
"v1": 1
73+
"R: pgadmins"
74+
"K: PGAdmin"
75+
"v1beta1": 1
76+
"R: pgupgrades"
77+
"K: PGUpgrade"
78+
"v1beta1": 1
79+
```
80+
-->
81+
82+
Every Kubernetes API request includes the Group, Resource, Version, and Kind of its payload and expected response.
83+
The version affects how Kubernetes handles the request, but it does *not* affect how Kubernetes stores the result.
84+
Every Kubernetes [object] is stored according to its Group, Resource, Namespace, and Name.
85+
86+
> [!NOTE]
87+
> - The API request URL contains the Group + Version + Resource (GVR).
88+
> - The API request body includes the Group + Version (GV) as [`apiVersion`] and Kind (K) as `kind`.
89+
> - [RBAC] matches on the Group + Resource (GR) of an API request.
90+
> - The etcd key of each object contains the Group + Resource (GR), Namespace and Name.
91+
92+
This allows a variety of clients to concurrently use whichever API versions they understand.
93+
Kubernetes converts what is stored to or from the version in the API request.
94+
This means, however, that *every* version of a resource **must** be equivalent *every other* version.
95+
96+
Each CRD indicates which versions Kubernetes should accept from clients with `served=true`.
97+
Kubernetes stores custom resource objects in the *single* version indicated with `storage=true`.
98+
99+
> [!IMPORTANT]
100+
> We use the `None` conversion strategy and [validation ratcheting](validation.md#validation-ratcheting)...
101+
102+
[`apiVersion`]: https://docs.k8s.io/reference/using-api#api-groups
103+
[object]: https://docs.k8s.io/concepts/overview/working-with-objects
104+
[RBAC]: https://docs.k8s.io/reference/kubernetes-api/authorization-resources/role-v1
105+
106+
<!--
107+
```mermaid
108+
venn
109+
sets Group
110+
sets Kind
111+
sets Resource
112+
sets Version
113+
114+
sets Group,Resource label: RBAC
115+
sets Group,Resource label: request url
116+
sets Group,Version label: request body "apiVersion"
117+
sets Group,Version,Kind label: request body
118+
sets Group,Version,Kind label: response body
119+
sets Group,Version,Resource label: storage
120+
```
121+
-->

‎pkg/apis/postgres-operator.crunchydata.com/validation.md‎

Lines changed: 89 additions & 8 deletions
Original file line numberDiff line numberDiff line change
@@ -4,18 +4,17 @@
44
# SPDX-License-Identifier: Apache-2.0
55
-->
66

7-
# Custom Resource Definitions
7+
# Custom Resource Definition Schemas
88

9-
These directories contain Go types that serve as [DTO]s for communicating with the Kubernetes API.
10-
We use the [controller-gen] tool to produce [CRD]s with schemas that match the Go types.
9+
These directories contain Go types that [controller-gen] uses to generate matching [CRD] schemas.
1110
The CRD schema tells Kubernetes what fields and values are allowed in our API objects and how to handle changes to values.
1211

1312
> [!TIP]
1413
> The CRD schema is *most* of the UX of our API
1514
1615
CRD schemas are modified OpenAPI 3.0 [validation] schemas.
1716
Much of the schema defines what fields, types, and values are *allowed*.
18-
`controller-gen` considers the [Go type] of a field and its [validation markers] for this.
17+
`controller-gen` considers the field's [Go type] and [validation markers] for this.
1918

2019
Kubernetes uses its own algorithm to consider and accept changes to API objects: [Server-Side Apply], SSA.
2120
CRD schemas contain non-standard attributes that affect SSA.
@@ -25,9 +24,6 @@ CRD schemas contain non-standard attributes that affect SSA.
2524

2625
[controller-gen]: https://book.kubebuilder.io/reference/controller-gen
2726
[CRD]: https://docs.k8s.io/tasks/extend-kubernetes/custom-resources/custom-resource-definitions
28-
[DTO]: https://martinfowler.com/eaaCatalog/dataTransferObject.html
29-
[Go type]: https://go.dev/ref/spec#Types
30-
[Kubernetes API]: https://docs.k8s.io/concepts/overview/kubernetes-api
3127
[processing markers]: https://book.kubebuilder.io/reference/markers/crd-processing
3228
[Server-Side Apply]: https://docs.k8s.io/reference/using-api/server-side-apply
3329
[validation]: https://docs.k8s.io/tasks/extend-kubernetes/custom-resources/custom-resource-definitions#validation
@@ -92,7 +88,7 @@ The `additionalProperties` property indicates that the keys are unknown; these f
9288
# CEL Rules
9389

9490
> [!IMPORTANT]
95-
> When possible, use [OpenAPI properties](#FIXME) rather than CEL rules.
91+
> When possible, use [OpenAPI properties](#openapi-properties) rather than CEL rules.
9692
> The former do not affect the CRD [validation budget](#FIXME). <!-- https://imgur.com/CzpJn3j -->
9793
9894
## Optional field syntax
@@ -109,3 +105,88 @@ likewise be considered optional.
109105
The optional field syntax is only available in K8s 1.29+.
110106

111107
[optional field marker]: https://pkg.go.dev/github.com/google/cel-go/cel#hdr-Syntax_Changes-OptionalTypes.
108+
109+
## CEL Availability
110+
111+
Kubernetes' capabilities with CEL are continuously expanding.
112+
Different versions of Kubernetes have different CEL functions, syntax, and features.
113+
114+
```asciidoc
115+
:controller-tools: https://github.com/kubernetes-sigs/controller-tools/releases
116+
117+
[cols=",,", options="header"]
118+
|===
119+
| Kubernetes | OpenShift | `controller-gen`
120+
121+
| 1.25 Beta, `CustomResourceValidationExpressions` gate
122+
| OCP 4.12
123+
| link:{controller-tools}/v0.9.0[v0.9.0] has `rule` and `message` fields on the `XValidation` marker
124+
125+
| 1.27 adds `messageExpression`
126+
| OCP 4.14
127+
| link:{controller-tools}/v0.15.0[v0.15.0] adds `messageExpression` field to the `XValidation` marker
128+
129+
| 1.28 adds `reason` and `fieldPath`
130+
| OCP 4.15
131+
| link:{controller-tools}/v0.16.0[v0.16.0] adds `reason` and `fieldPath` to the `XValidation` marker
132+
133+
| 1.29 GA | OCP 4.16 |
134+
135+
| 1.30 enables link:#validation-ratcheting[validation ratcheting]; link:https://pr.k8s.io/123475[fixes fieldPath]...
136+
| OCP 4.17
137+
| link:{controller-tools}/v0.17.3[v0.17.3] adds `optionalOldSelf` to the `XValidation` marker
138+
139+
| 1.34 link:https://pr.k8s.io/132837[fixes IntOrString cost]
140+
| ?
141+
| link:{controller-tools}/v0.18.0[v0.18.0] allows validation on IntOrString
142+
143+
| 1.35 link:https://pr.k8s.io/132798[shows values when validation fails]
144+
| ?
145+
| n/a
146+
147+
|===
148+
```
149+
150+
<!-- TODO: long-form; describe each library -->
151+
152+
Some details are missing from the Go package documentation: https://pr.k8s.io/130660
153+
154+
| CEL [libraries](https://code.k8s.io/staging/src/k8s.io/apiserver/pkg/cel/library), extensions, etc. | Kubernetes | OpenShift |
155+
| --- | --- | --- |
156+
| kubernetes.authz | 1.28 |
157+
| kubernetes.authzSelectors | 1.32 |
158+
| kubernetes.format | 1.32 | [4.18](https://github.com/openshift/kubernetes/pull/2140) |
159+
| kubernetes.lists | 1.24 | 4.12 |
160+
| kubernetes.net.cidr | 1.31 | [4.16](https://github.com/openshift/kubernetes/pull/1828) |
161+
| kubernetes.net.ip | 1.31 | [4.16](https://github.com/openshift/kubernetes/pull/1828) |
162+
| kubernetes.quantity | 1.29 | 4.16 |
163+
| kubernetes.regex | 1.24 | 4.12 |
164+
| kubernetes.urls | 1.24 | 4.12 |
165+
| [cross-type numeric comparison](https://pkg.go.dev/github.com/google/cel-go/cel#CrossTypeNumericComparisons) | 1.29 | 4.16 |
166+
| [optional types](https://pkg.go.dev/github.com/google/cel-go/cel#OptionalTypes) | 1.29 | 4.16 |
167+
| [strings](https://pkg.go.dev/github.com/google/cel-go/ext#Strings) v0 | 1.24 | 4.12 |
168+
| [strings](https://pkg.go.dev/github.com/google/cel-go/ext#Strings) v2 | 1.30 | 4.17 |
169+
| [sets](https://pkg.go.dev/github.com/google/cel-go/ext#Sets) | 1.30 | 4.17 |
170+
| [two-variable comprehension](https://pkg.go.dev/github.com/google/cel-go/ext#TwoVarComprehensions) | 1.33 |
171+
172+
173+
# Validation Ratcheting
174+
175+
> **Feature Gate:** `CRDValidationRatcheting`
176+
>
177+
> Enabled in Kubernetes 1.30 and GA in 1.33 (OpenShift 4.17 and ~4.20)
178+
179+
[Validation ratcheting] allows update operations to succeed when unchanged fields are invalid.
180+
This allows CRDs to add or "tighten" validation without breaking existing CR objects.
181+
182+
Some schema changes are not ratcheted:
183+
184+
- OpenAPI `allOf`, `oneOf`, `anyOf`, `not`; values in fields with these must be valid
185+
- OpenAPI `required`; required fields are always required
186+
- Removing `additionalProperties`; undefined fields are always dropped
187+
- Adding or removing fields (names) in `properties`; undefined fields are dropped, and values in new fields must be valid
188+
- Changes to `x-kubernetes-list-type` or `x-kubernetes-list-map-keys`; values in these fields must be valid
189+
- Rules containing `oldSelf`; these are [transition rules] and should do their own ratcheting
190+
191+
[transition rules]: https://docs.k8s.io/tasks/extend-kubernetes/custom-resources/custom-resource-definitions#transition-rules
192+
[Validation ratcheting]: https://docs.k8s.io/tasks/extend-kubernetes/custom-resources/custom-resource-definitions#validation-ratcheting

0 commit comments

Comments
(0)

AltStyle によって変換されたページ (->オリジナル) /