-
Notifications
You must be signed in to change notification settings - Fork 634
chore: add chainsaw e2e tests support #3844
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Draft
Draft
Changes from all commits
Commits
Show all changes
6 commits
Select commit
Hold shift + click to select a range
0f884e1
chore: add chainsaw e2e tests support
eddycharly 3c7ce92
Merge branch 'master' into chainsaw
eddycharly ed78245
bump version
eddycharly 3c64ba2
regen
eddycharly f1c2e0c
generate
eddycharly 66bacd8
revet env vars changes
eddycharly File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
The table of contents is too big for display.
Diff view
Diff view
There are no files selected for viewing
13 changes: 13 additions & 0 deletions
.chainsaw.yaml
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,13 @@ | ||
| # yaml-language-server: $schema=https://raw.githubusercontent.com/kyverno/chainsaw/main/.schemas/json/configuration-chainsaw-v1alpha1.json | ||
| apiVersion: chainsaw.kyverno.io/v1alpha1 | ||
| kind: Configuration | ||
| metadata: | ||
| name: configuration | ||
| spec: | ||
| parallel: 2 | ||
| timeouts: | ||
| assert: 5m0s | ||
| cleanup: 5m0s | ||
| delete: 5m0s | ||
| error: 5m0s | ||
| exec: 5m0s |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
1 change: 1 addition & 0 deletions
.gitignore
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -1,3 +1,4 @@ | ||
| .DS_Store | ||
| /vendor/ | ||
| /testing/chainsaw/e2e-generated*/ | ||
| /testing/kuttl/e2e-generated*/ |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
92 changes: 92 additions & 0 deletions
testing/chainsaw/README.md
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,92 @@ | ||
| # KUTTL | ||
|
|
||
| ## Installing | ||
|
|
||
| Docs for install: https://kuttl.dev/docs/cli.html#setup-the-kuttl-kubectl-plugin | ||
|
|
||
| Options: | ||
| - Download and install the binary | ||
| - Install the `kubectl krew` [plugin manager](https://github.com/kubernetes-sigs/krew) | ||
| and `kubectl krew install kuttl` | ||
|
|
||
| ## Cheat sheet | ||
|
|
||
| ### Suppressing Noisy Logs | ||
|
|
||
| KUTTL gives you the option to suppress events from the test logging output. To enable this feature | ||
| update the `kuttl` parameter when calling the `make` target | ||
|
|
||
| ``` | ||
| KUTTL_TEST='kuttl test --suppress-log=events' make check-kuttl | ||
| ``` | ||
|
|
||
| To suppress the events permanently, you can add the following to the KUTTL config (kuttl-test.yaml) | ||
| ``` | ||
| suppress: | ||
| - events | ||
| ``` | ||
|
|
||
| ### Run test suite | ||
|
|
||
| Make sure that the operator is running in your Kubernetes environment and that your `kubeconfig` is | ||
| set up. Then run the make targets: | ||
|
|
||
| ``` | ||
| make generate-kuttl check-kuttl | ||
| ``` | ||
|
|
||
| ### Running a single test | ||
|
|
||
| A single test is considered to be one directory under `kuttl/e2e-generated`, for example | ||
| `kuttl/e2e-generated/restore` is the `restore` test. | ||
|
|
||
| There are two ways to run a single test in isolation: | ||
| - using an env var with the make target: `KUTTL_TEST='kuttl test --test <test-name>' make check-kuttl` | ||
| - using `kubectl kuttl --test` flag: `kubectl kuttl test testing/kuttl/e2e-generated --test <test-name>` | ||
|
|
||
| ### Writing additional tests | ||
|
|
||
| To make it easier to read tests, we want to put our `assert.yaml`/`errors.yaml` files after the | ||
| files that create/update the objects for a step. To achieve this, infix an extra `-` between the | ||
| step number and the object/step name. | ||
|
|
||
| For example, if the `00` test step wants to create a cluster and then assert that the cluster is ready, | ||
| the files would be named | ||
|
|
||
| ```yaml | ||
| 00--cluster.yaml # note the extra `-` to ensure that it sorts above the following file | ||
| 00-assert.yaml | ||
| ``` | ||
|
|
||
| ### Generating tests | ||
|
|
||
| KUTTL is good at setting up K8s objects for testing, but does not have a native way to dynamically | ||
| change those K8s objects before applying them. That means that, if we wanted to write a cluster | ||
| connection test for PG 13 and PG 14, we would end up writing two nearly identical tests. | ||
|
|
||
| Rather than write those multiple tests, we are using `envsubst` to replace some common variables | ||
| and writing those files to the `testing/kuttl/e2e-generated*` directories. | ||
|
|
||
| These templated test files can be generated by setting some variables in the command line and | ||
| calling the `make generate-kuttl` target: | ||
|
|
||
| ```console | ||
| KUTTL_PG_VERSION=13 KUTTL_POSTGIS_VERSION=3.0 make generate-kuttl | ||
| ``` | ||
|
|
||
| This will loop through the files under the `e2e` and `e2e-other` directories and create matching | ||
| files under the `e2e-generated` and `e2e-generated-other` directories that can be checked for | ||
| correctness before running the tests. | ||
|
|
||
| Please note, `make check-kuttl` does not run the `e2e-other` tests. To run the `postgis-cluster` | ||
| test, you can use: | ||
|
|
||
| ``` | ||
| kubectl kuttl test testing/kuttl/e2e-generated-other/ --timeout=180 --test postgis-cluster` | ||
| ``` | ||
|
|
||
| To run the `gssapi` test, please see testing/kuttl/e2e-other/gssapi/README.md. | ||
|
|
||
| To prevent errors, we want to set defaults for all the environment variables used in the source | ||
| YAML files; so if you add a new test with a new variable, please update the Makefile with a | ||
| reasonable/preferred default. |
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.