License Release Artifact Hub GitHub Actions Build Java Repo Size
Apache SparkTM K8s Operator is a subproject of Apache Spark and aims to extend K8s resource manager to manage Apache Spark applications via Operator Pattern.
Apache Spark provides a Helm Chart.
- https://apache.github.io/spark-kubernetes-operator/
- https://artifacthub.io/packages/helm/spark-kubernetes-operator/spark-kubernetes-operator/
helm repo add spark https://apache.github.io/spark-kubernetes-operator helm repo update helm install spark spark/spark-kubernetes-operator
Spark K8s Operator is built using Gradle. To build, run:
./gradlew build -x test./gradlew build
When upgrading dependencies, regenerate gradle/verification-metadata.xml from scratch.
Since --write-verification-metadata is append-only and keeps stale entries of removed or
old dependencies, delete the existing file first so that the regenerated file contains
only the checksums of the current dependencies. Pass --refresh-dependencies so that
metadata files (BOM / parent POMs) already present in the local Gradle cache are
downloaded and recorded again — without it their checksums are silently dropped and
CI fails with a cold cache:
rm gradle/verification-metadata.xml ./gradlew --write-verification-metadata sha512 build --refresh-dependencies
Note that Gradle rewrites the file without the ASF license header, so restore the header after regeneration.
./gradlew buildDockerImage
helm install spark -f build-tools/helm/spark-kubernetes-operator/values.yaml build-tools/helm/spark-kubernetes-operator/
$ kubectl apply -f examples/pi.yaml $ kubectl get sparkapp NAME CURRENT STATE AGE pi ResourceReleased 4m10s $ kubectl delete sparkapp/pi
$ kubectl apply -f examples/prod-cluster-with-three-workers.yaml $ kubectl get sparkcluster NAME CURRENT STATE AGE prod RunningHealthy 10s $ kubectl port-forward prod-master-0 6066 & $ ./examples/submit-pi-to-prod.sh { "action" : "CreateSubmissionResponse", "message" : "Driver successfully submitted as driver-20260110030233-0000", "serverSparkVersion" : "4.2.0", "submissionId" : "driver-20260110030233-0000", "success" : true } $ curl http://localhost:6066/v1/submissions/status/driver-20260110030233-0000/ { "action" : "SubmissionStatusResponse", "driverState" : "FINISHED", "serverSparkVersion" : "4.2.0", "submissionId" : "driver-20260110030233-0000", "success" : true, "workerHostPort" : "10.1.1.172:44233", "workerId" : "worker-20260110030145-10.1.1.172-44233" } $ kubectl delete sparkcluster prod sparkcluster.spark.apache.org "prod" deleted
If you have not yet done so, follow YuniKorn docs to install the latest version:
helm repo add yunikorn https://apache.github.io/yunikorn-release helm repo update helm install yunikorn yunikorn/yunikorn --namespace yunikorn --version 1.9.0 --create-namespace --set embedAdmissionController=false
Submit a Spark app to YuniKorn enabled cluster:
$ kubectl apply -f examples/pi-on-yunikorn.yaml $ kubectl describe pod pi-on-yunikorn-0-driver ... Events: Type Reason Age From Message ---- ------ ---- ---- ------- Normal Scheduling 1s yunikorn default/pi-on-yunikorn-0-driver is queued and waiting for allocation Normal Scheduled 1s yunikorn Successfully assigned default/pi-on-yunikorn-0-driver to node docker-desktop Normal PodBindSuccessful 1s yunikorn Pod default/pi-on-yunikorn-0-driver is successfully bound to node docker-desktop Normal Pulled 0s kubelet Container image "apache/spark:4.2.0-scala" already present on machine Normal Created 0s kubelet Created container: spark-kubernetes-driver Normal Started 0s kubelet Started container spark-kubernetes-driver $ kubectl delete sparkapp pi-on-yunikorn sparkapplication.spark.apache.org "pi-on-yunikorn" deleted from default namespace
Check the existing Spark applications and clusters. If exists, delete them.
$ kubectl get sparkapp No resources found in default namespace. $ kubectl get sparkcluster No resources found in default namespace.
Remove HelmChart and CRDs.
helm uninstall spark kubectl delete crd sparkapplications.spark.apache.org kubectl delete crd sparkclusters.spark.apache.org
Please review the Contribution to Spark guide for information on how to get started contributing to the project.