-
Notifications
You must be signed in to change notification settings - Fork 246
Discuss adding a Docker-based workflow for running tests locally #1682
Hi Cloudberry contributors,
I am new to this project and recently tried to run Apache Cloudberry locally to better understand it.
I was able to start the project using the Docker sandbox and connect to the database successfully. That worked well for trying the database manually. After that, I wanted to go one step further and run the project tests
locally.
From what I found, the sandbox containers are mainly useful for running a local Cloudberry cluster, but they do not seem to include the source/build tree or the full test environment needed for commands like:
make -C src/test/regress installcheck-small
After looking at the GitHub Actions workflow, it seems that CI uses a different approach: a build image, source checkout, configure/build/install steps, creation of a gpAux/gpdemo cluster, and then make installcheck-* targets.
I was able to reproduce a similar workflow locally by using the official build image:
apache/incubator-cloudberry:cbdb-build-rocky9-latest
and mounting the source tree into the container. After building Cloudberry and creating a demo cluster, I was able to run:
PGOPTIONS="-c optimizer=off" make -C src/test/regress installcheck-small
successfully.
This could lower the entry barrier for new contributors who want not only to run Cloudberry locally, but also to validate changes before opening a pull request.
I would like to discuss whether it would be useful to make this workflow easier for new contributors.
Possible options:
- Add test support to the existing Docker sandbox, if that fits its purpose.
- Add a separate development/test Docker container or documented workflow, similar to what CI does.
- Add documentation to src/test/README, devops/sandbox/README.md, or the contributor guide explaining how to run tests locally with Docker.
I understand that the sandbox may not be intended as a full development/test environment, so a separate test-oriented container or documented workflow might be a better fit.
Would the project be open to adding something like this? If so, what direction would maintainers prefer?
All reactions
-
👍 1
Great suggestion!
I think we could add your approach to the /devops/README.md directory at https://github.com/apache/cloudberry/tree/main/devops as well as to the documentation at https://cloudberry.apache.org/docs/deployment/build-based-on-docker. This would make it more convenient for contributors and developers to leverage the existing development image for testing code changes. How about this?
If it sounds good, could you help create the pull request for it? Thanks!
Replies: 1 comment 2 replies
We could add something similar to the readme or make the creation more automated. What do you think of this idea?
Local Test Container
You can run tests against a local checkout without installing build
dependencies on the host by using the published Cloudberry build image and a
bind mount. The source tree remains on the host, so edits made on the host or
inside the container affect the same files.
From the repository root:
docker run -dit \ --privileged \ --user root \ --hostname cdw \ --name cloudberry-devtest \ --shm-size=2gb \ --ulimit core=-1 \ --cgroupns=host \ -v /sys/fs/cgroup:/sys/fs/cgroup:rw \ -v "$PWD":/workspace/cloudberry \ -w /workspace/cloudberry \ apache/incubator-cloudberry:cbdb-build-rocky9-latest \ bash -lc "sleep infinity"
Enter the container:
docker exec -it --user gpadmin -w /workspace/cloudberry cloudberry-devtest bashPrepare the build and demo cluster inside the container:
export SRC_DIR=/workspace/cloudberry export BUILD_DESTINATION=/usr/local/cloudberry-db /tmp/init_system.sh ./devops/build/automation/cloudberry/scripts/configure-cloudberry.sh ./devops/build/automation/cloudberry/scripts/build-cloudberry.sh ./devops/build/automation/cloudberry/scripts/create-cloudberry-demo-cluster.sh
After changing code, rebuild the relevant parts or rerun the build script, then
run the tests you need. For example:
source /usr/local/cloudberry-db/cloudberry-env.sh source /workspace/cloudberry/gpAux/gpdemo/gpdemo-env.sh PGOPTIONS="-c optimizer=off" make -C src/test/regress installcheck-small PGOPTIONS="-c optimizer=off" make -C src/test/regress installcheck-tests TESTS="insert" make -C src/test/isolation2 installcheck-isolation2
Stop or remove the container when it is no longer needed:
docker stop cloudberry-devtest docker rm -f cloudberry-devtest
All reactions
welcome to comment on #1692.
All reactions
Great suggestion!
I think we could add your approach to the /devops/README.md directory at https://github.com/apache/cloudberry/tree/main/devops as well as to the documentation at https://cloudberry.apache.org/docs/deployment/build-based-on-docker. This would make it more convenient for contributors and developers to leverage the existing development image for testing code changes. How about this?
If it sounds good, could you help create the pull request for it? Thanks!