Sourced from ray's releases.

Ray-2.54.0

Ray Data

🎉 New Features

• Add checkpointing support to Ray Data (#59409)
• Compute Expressions: list operations (#59346), fixed-size arrays (#58741), string padding (#59552), logarithmic (#59549), trigonometric (#59712), arithmetic (#59678), and rounding (#59295)
• Add sql_params support to read_sql (#60030)
• Add AsList aggregation (#59920)
• Support CountDistinct aggregate (#59030)
• Add credential provider abstraction for Databricks UC datasource (#60457)
• Support callable classes for UDFExpr (#56725)
• Add autoscaler metrics to Data Dashboard (#60472)
• Add optional filesystem parameter to download expression (#60677)
• Allow specifying partitioning style or flavor in write_parquet() (#59102)
• New cluster autoscaler enabled by default (#60474)

💫 Enhancements

• Improve numerical stability in scalers by handling near-zero values (#60488)
• Export dataset operator output schema to event logger (#60086)
• Iceberg: add retry policy for Storage + Catalog writes (#60620)
• Iceberg: remove calls to Catalog Table in write tasks (#60476)
• Expose logical operators and rules via package exports (#60297, #60296)
• Demote Sort from requiring preserve_order (#60555)
• Improve appearance of repr(dataset) (#59631)
• Allow configuring DefaultClusterAutoscalerV2 thresholds via env vars (#60133)
• Use Arrow IPC for Arrow Schema serialization/deserialization (#60195)
• Store _source_paths in object store to prevent excessive spilling during read task serialization (#59999)
• Add more shuffle fusion rules (#59985)
• Enable and tune DownstreamCapacityBackpressurePolicy (#59753)
• Enable concurrency cap backpressure with tuning (#59392)
• Set default actor pool scale up threshold to 1.75 (#59512)
• Don't downscale actors if the operator hasn't received any inputs (#59883)
• Don't reserve GPU budget for non-GPU tasks (#59789)
• Only return selected data columns in hive-partitioned Parquet files (#60236)
• Ordered + FIFO bundle queue (#60228)
• Add node_id, pid, attempt number for hanging tasks (#59793)
• Revise resource allocator task scheduling to factor in pending task outputs (#60639)
• Track block serialization time (#60574)
• Use metrics from OpRuntimeMetrics for progress (#60304)
• Tabular form for streaming executor op metrics (#59774)
• Info-log cluster scale-up decisions (#60357)
• Use plain mode instead of grid mode for OpMetrics logging (#59907)
• Progress reporting refactors (#59350, #59629, #59880)
• Remove deprecated TENSOR_COLUMN_NAME constant (#60573)
• Remove meta_provider parameter (#60379)
• Decouple Ray Train from Ray Data by removing top-level ray.data imports (#60292)
• Move extension types to ray.data (#59420)
• Skip upscaling validation warning for fixed-size actor pools (#60569)

... (truncated)

• 48bd1f8 [data] revert "continue grabbing task state until response is not None" (#61066)
• 165b4aa [Data][Cherry-pick] Fixed min_scheduling_resources to fallback to incremental...
• 6835277 [rllib] disable failing tests that are not release blocking (#60933)
• f8e1102 [Core] Fix test_failed_task_runtime_env_setup failure on windows (#60852) (#6...
• 620214f [RLlib] Fix _test_dependency_torch (#60742) (#60888)
• 5d2115c cherrypick part of #60887 (#60897)
• e94871d [Data][Cherry-pick] Prevent Limit from getting pushed past map_groups (#6...
• a77457b [Data] Fixing ReservationOpResourceAllocator to properly borrow resources f...
• 0f27a3a [Serve] Video analysis example fix (#60784) (#60871)
• 1b1a9bd [Data] Fixing output backpressure unblocking sequence to properly handle term...
• Additional commits viewable in compare view

You can trigger Dependabot actions by commenting on this PR:

• @dependabot rebase will rebase this PR
• @dependabot recreate will recreate this PR, overwriting any edits that have been made to it
• @dependabot show <dependency name> ignore conditions will show all of the ignore conditions of the specified dependency
• @dependabot ignore this major version will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself)
• @dependabot ignore this minor version will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself)
• @dependabot ignore this dependency will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself)
  You can disable automated security fix PRs for this repo from the Security Alerts page.