Key updates in these image versions include:
2.3 image version, 2.3 clusters with Lightning Engine now use Iceberg version 1.10 by default.2.2 and 2.3 image versions:
1.49 to support lineage for tables created using the Lakehouse Runtime catalog.Managed Service for Apache Spark (formerly Dataproc on Compute Engine): Fixed a segmentation fault when OpenLineage parses complex SQL query strings.
]]>Key updates in these runtime versions include:
2.3 runtime:
1.49 to support lineage for tables created using the Lakehouse Runtime catalog.Managed Service for Apache Spark (formerly Dataproc on Compute Engine):
A critical bug related to Conda channels has been fixed in-place in image versions
1.4.81, 1.5.92, 2.1.117, and 2.2.85. These image versions were released without pre-configured Conda channels.
Required customer actions: To comply with Google requirements, recreate the following resources if they were created using these image versions on or before August 10, 2026:
New Managed Service for Apache Spark (formerly Google Cloud Serverless for Apache Spark) subminor runtime versions:
Notes:
Apache Spark upgraded to 3.5.3 in 2.2 runtime.
Apache Gluten upgraded to 1.6 in 2.3 runtime.
Managed Service for Apache Spark latest image and runtime versions:
spark.scheduler.listenerbus.exitTimeout to 30s.Key updates in this image version include:
3.5-dataproc-28 subminor image version doesn't have preconfigured Conda channels, and is mapped to default aliases (such as 3.5 and latest).
3.5-dataproc-28 or using default aliases (3.5, latest), packages cannot be installed using Conda unless channels are manually configured during cluster initialization.You may need to delete and replace existing clusters After August 25, 2026, existing clusters created with images that have preconfigured Conda channels (even if cluster jobs don't use Conda to install packages) need to be deleted and replaced with new clusters created or recreated with images that don't have preconfigured Conda channels.
]]>Starting with Managed Service for Apache Spark image version 3.0,
clusters that are created without a specified machine type for a node are created with a
Flex VM
configuration for the node.
Announcing the General Availability (GA) release of Managed Service for Apache Spark cluster
image version 3.0,
as follows:
3.0.0-debian13, 3.0.0-ml-ubuntu24, 3.0.0-rocky9, and 3.0.0-ubuntu24.
3.0.0-ml-ubuntu24 image extends the 3.0 base image with ML-specific libraries.Image version 3.0 is a lightweight image that contains the following
pre-installed core components, reducing exposure to Common Vulnerabilities and Exposures (CVEs):
You can add
listed optional components
when you create a 3.0 image version cluster.
Recommendation: Use the 3.0 (and later) image versions to meet security
compliance requirements.
Managed Service for Apache Spark (formerly Dataproc on Compute Engine):
2.1, 2.2 and 2.3 cluster image versions now support
Confidential Compute
for the
g4-standard-48
GPU machine type.Managed Service for Apache Spark (formerly Google Cloud Serverless for Apache Spark):
The 3.0 runtime now uses fewer executors, as follows:
spark.dynamicAllocation.minExecutors propertyspark.executor.instances and spark.dynamicAllocation.initialExecutors propertiesAll runtimes now configure spark.scheduler.listenerbus.exitTimeout to 30 seconds.
Key updates in these image versions include:
2.1.117 and 2.2.85 subminor image versions don't have preconfigured Conda channels, and are not mapped to default aliases (such as 2.1-debian11 and 2.2-debian12) until August 25, 2026.
2.1.117-debian11 or 2.2.85-debian12). Packages cannot be installed using Conda unless channels are manually configured during cluster initialization.2.1.116-debian11 or 2.2.84-debian12).2.1 and 2.2 will point to image versions without preconfigured Conda channels (the latest subminor versions). All workloads must transition to these new images after August 25, 2026 since the use of prior subminor versions with preconfigured Conda channels will be disallowed.You may need to delete and replace existing clusters After August 25, 2026, existing clusters created with images that have preconfigured Conda channels (even if cluster jobs don't use Conda to install packages) need to be deleted and replaced with new clusters created or recreated with images that don't have preconfigured Conda channels.
]]>Managed Service for Apache Spark (formerly Dataproc on Compute Engine):
The following new subminor image versions, 1.3.96, 1.4.81, 1.5.92, 2.0.161, and 2.3.32,
don't have preconfigured Conda channels, and are not mapped to default aliases
(such as 2.3-debian12 and 2.3-ubuntu22) until August 25, 2026.
The new 2.1 and 2.2 subminor images continue to have preconfigured Conda channels,
but note the following Default change schedule:, which will impact all supported
image versions.
2.3.32-debian12). Packages cannot be installed using Conda unless channels are manually configured during cluster initialization.1.3.96, 1.4.81, 1.5.92, and 2.0.161 will become default after August 25, 2026.
Additionally, newer subminor versions for 2.1, 2.2, and 2.3 released after August 25, 2026 will not have preconfigured Conda channels and will be mapped to default aliases.
All workloads must use new images after August 25, 2026
since the use of prior subminor versions with preconfigured Conda
channels will be disallowed.You may need to delete and replace existing clusters After August 25, 2026, existing clusters created with images that have preconfigured Conda channels (even if cluster jobs don't use Conda to install packages) need to be deleted and replaced with new clusters created or recreated with images that don't have preconfigured Conda channels.
Managed Service for Apache Spark (formerly Dataproc on Compute Engine): Key updates in these image versions include:
ICEBERG optional component. Users can opt-in to Iceberg 1.10 by setting the dataproc:dataproc.iceberg.version cluster property to 1.10 during cluster creation.spark.scheduler.listenerbus.exitTimeout to 30s for Compute Engine deployments.Managed Service for Apache Spark (formerly Dataproc on Compute Engine): Rollout of the new sub-minor versions without pre-configured channels will begin on June 22, 2026, delayed from the previously planned date of June 15, 2026 ETA.
]]>Key updates in these runtime versions include:
Managed Service for Apache Spark (formerly Dataproc on Compute Engine):
Added support for selecting specific Confidential Computing technologies (AMD SEV, AMD SEV-SNP, Intel TDX) when creating clusters using the new --confidential-compute-type flag in gcloud and the confidentialInstanceType field in the API. The boolean --enable-confidential-compute flag is now deprecated but will continue to function, defaulting to AMD SEV for backward compatibility.
confidentialInstanceType enum in the API.--enable-confidential-compute flag and enableConfidentialCompute field are deprecated in favor of the new type-specific flag/field.SEV.SEV, SEV-SNP, and TDX.Managed Service for Apache Spark (formerly Dataproc on Compute Engine): The following subminor image versions announced on May 19, 2026 have been rolled back:
Rollback Notice: The 2.2.82 image versions were rolled back on May 25, 2026.
Managed Service for Apache Spark (formerly Dataproc on Compute Engine):
The configuration for Spark shuffle partitions (spark.sql.shuffle.partitions) has changed from an integer to a string type.
This change impacts image versions 2.3.30 and later in version 2.3, and 2.2.82 and later in version 2.2.
spark.conf.set() with an integer literal.
spark.conf.set("spark.sql.shuffle.partitions", 100)spark.conf.set("spark.sql.shuffle.partitions", "100")spark-submit --conf spark.sql.shuffle.partitions=100), properties files, or Spark SQL commands (spark.sql("SET spark.sql.shuffle.partitions=100")) remains unaffected, as these methods naturally parse the input as strings.New Managed Service for Apache Spark (formerly Dataproc on Compute Engine) subminor cluster image versions for 3.0 (Preview):
Key updates in this release:
Key updates in this release:
v0.0.79.26.04.0 version in the Managed Service
for Apache Spark version 3.0 serverless runtime.Key updates in this release:
Managed Service for Apache Spark (formerly Dataproc on Compute Engine): Announcing the following cluster features:
Dataproc and Google Cloud Serverless for Apache Spark are now unified in the Google Cloud console under the Managed Service for Apache Spark brand. This change consolidates our managed Spark deployment options into a single umbrella brand that includes the full breadth of our Spark capabilities. No existing functionality is being removed as part of this change, and there will be no impact to the Dataproc API, client library, gcloud CLI, or Dataproc IAM role and permission names.
]]>3.9.5 in image version 2.3.New Serverless for Apache Spark runtime versions:
New Dataproc on Compute Engine subminor image versions:
New Serverless for Apache Spark runtime versions:
New Dataproc on Compute Engine subminor image versions:
2.1, 2.2 and 2.3 images.New Serverless for Apache Spark runtime versions:
Dataproc on Compute Engine: The following subminor image versions announced on March 08, 2026 have been rolled back:
New Dataproc on Compute Engine subminor image versions:
Rollback Notice: These image versions were rolled back on March 11, 2026.
New Serverless for Apache Spark runtime versions:
Serverless for Apache Spark 3.0 runtime version:
Serverless for Apache Spark 1.2, 2.2 and 2.3 runtime versions: