Bolt has improved it by treating Shuffle as an operator in Bolt and offloading it from Gluten to Bolt for parallel processing.

@frankobe would you give more explanation about treat shuffle as an operator? doest Bolt still fetch shuffle data from spark's java iterator?

When shuffle writer or shuffle reader is adjacent to other operators that run in Gluten,it would be offloaded into Bolt and treated as a Bolt operator.

For the shuffle reader, Bolt would fetch the shuffle data directly from spark's InputStream, and then read raw bytes from it and do decompression and deserialization inside the shuffle reader operator, and then outputs result as a normal operator instead using Iterator[ColumnarBatch].

For the shuffle writer, Bolt also constructs a shuffle writer operator. It receives data directly from the upstream operator instead of wrapping it into an Iterator[ColumnarBatch].

@zhangxffff thanks for the explanation!

For the shuffle reader

I'm curious about Morsel Driven mentioned above, I guess Bolt use parallel mode to execute task (Gluten use serial mode).
does fetching raw shuffle data, decompression, deserialization all done in one operator? how many driver does shuffle read operator generate? if only one driver of shuffle read op, seems no major difference with Gluten, if two, fetching, decompression, deserialization all in one driver?

@zhangxffff thanks for the explanation!

For the shuffle reader

I'm curious about Morsel Driven mentioned above, I guess Bolt use parallel mode to execute task (Gluten use serial mode). does fetching raw shuffle data, decompression, deserialization all done in one operator? how many driver does shuffle read operator generate? if only one driver of shuffle read op, seems no major difference with Gluten, if two, fetching, decompression, deserialization all in one driver?

After offload as a Bolt operator, shuffle read support multiple drivers, each driver's shuffle reader get different InputStreams (since Gluten get an iterator of InputStream from ShuffleBlockFetcherIterator, which contains multiple InputStreams), so they can fetching data, decompression, deserialization in parallel.

Bolt can dynamically over-issue memory after reaching the off-heap configuration value based on the actual physical memory usage, thereby reducing the amount of spilled data, improving performance, and reducing the risk of OOM.

Looking forward to see this open-sourced — especially how it helps avoid OOM. We’ve been struggling with that a lot.

@boneanxs We have implemented all memory management functions in Bolt, so Bolt can dynamically over-issue memory after reaching the off-heap configuration value based on the actual physical memory usage. The specific code is here:
https://github.com/bytedance/bolt/blob/c9b1507b696edbae28d5133d6d512a1eac77e446/bolt/common/memory/sparksql/ExecutionMemoryPool.cpp#L347-L419

We will also refactor the memory module later to avoid shortcomings of the existing mechanism. All planning and discussions will be conducted in the bolt repo's issues.

@WangGuangxin "We have implemented all memory management functions in Bolt, so Bolt can dynamically over-issue memory after reaching the off-heap configuration value based on the actual physical memory usage" IIRC, you mean Bolt memory pool can allocate over the config limit based on the current measured RSS to avoid a single query OOM. Then does it mean this might increase the risks of server OOMs. If that's the case, the memory config limit is more like an optimization to avoid frequent RSS limit check?
For "refactor the memory module later to avoid shortcomings of the existing mechanism", do you have details for these? Thank you!

It's good idea since Flink doesn't have many customer adoptions yet.

@WangGuangxin i am interested in this timeline too.

@afterincomparableyum Bolt is compatible with Celeborn (which is used internally in ByteDance as well). We are planning a deeper integration with Celeborn C++ client for more flexibility & performance. Happy to collaborate in the community if you are interested

i’m happy to collaborate 😀

one more question @frankobe , i’m assuming that the gluten parameter for dynamic off heap sizing, spark.gluten.memory.dynamic.offHeap.sizing.enabled=true, works with Bolt too since it is a fork off of velox?

Or would we not need to specify off heap memory configs at all since Bolt has Memory Management Offload.

If all of these questions are going to be answered in docs for the PR you will submit, I can wait till those docs get released before asking more questions. Thank you!

The comparison with GPU/Flink support is not appropriate.
GPU and Flink support add new, non-overlapping capabilities to the ecosystem, whereas Bolt is essentially a downstream fork of Velox. These two cases are fundamentally different in nature, so drawing a direct analogy between them is misleading.

Gluten still needs to maintain its own use cases, and the proposed community support model raises concerns.
The idea of "assigning" community members sounds more like an internal work arrangement rather than participation driven by open-source interest. If those three members change roles or priorities in the future, this commitment may no longer hold. In that case, the sustainability assumption behind this support model would break. This makes the project feel closer to an internal corporate project, and it risks losing the essence of an open-source community.

The current contributor base of Bolt is highly concentrated within ByteDance.
At the moment, almost all open-source contributions to Bolt come from ByteDance. This situation reminds me of the ByteHouse and ClickHouse relationship: when an open-source compute engine fails to attract a broader and more diverse contributor base, it becomes problematic in the long run. I believe this would not be a positive outcome for the Gluten community either.

Gluten still needs to maintain its own use cases, and the proposed community support model raises concerns.
The idea of "assigning" community members sounds more like an internal work arrangement rather than participation driven by open-source interest. If those three members change roles or priorities in the future, this commitment may no longer hold. In that case, the sustainability assumption behind this support model would break. This makes the project feel closer to an internal corporate project, and it risks losing the essence of an open-source community.

It's actually from my suggestion. I was thinking to refer to current Clickhouse backend way that once we have some clickhouse issue during Velox development, especially CI issues, we mostly ping @zzcclp to solve or review. Similarly, to mostly ease velox and Clickhouse backend's development, I suggested Bolt backend also should have some guys we can ping to get help. The worst case is that if no one mentain Bolt backend one day, we will have to remove it or freeze it from Gluten. It's the same thing in Velox and Clickhouse backend actually. Like no one maintain velox backend, we will have to remove or freeze it.

Also please note there is no fundamental difference of Velox and Bolt, they are both corporate projects now, velox is even more closed as cooperate project that we can't get any real committers in no matter how much we contributed and how hard we tried. On the other side Bolt already WIP to submit Apache incubatoring request and build LF foundation on failure of Apache. Bolt welcome PRs from outside and would like to join as initial committers.

The current contributor base of Bolt is highly concentrated within ByteDance.

It's more like a chiken and egg issue. If Bolt isn't in, no one would like to contribute. Once bolt join the game with more maturality and openness, customer may move.

(...) velox is even more closed as cooperate project that we can't get any real committers in no matter how much we contributed and how hard we tried.

As a quick fact-check, there are 4-5 Velox maintainers involved in the Gluten community today, including one from ByteDance. The list is available here:

https://velox-lib.io/docs/community/components-and-maintainers

Both projects are following Apache 2.0 license, Bolt's dependency is declared in NOTICE.txt. We are here to address any specific legal concerns, though there are no known issues to our knowledge.

In terms of OSS license, yup, it's legal. But socially and technically weird.

What is bolt's future plan? A permanently independent fork with no intention to stay aligned long‐term? Or an upstream-first fork where new work goes upstream ASAP? Or a downstream first fork keeps cherry-picking without feedback?

Bolt backend #11261 only fork and modify the existing backends-velox (whose IP belongs to Gluten repository & ASF) without any Velox-related code

Yes, I also mean it's just a requirement for this patch. For velox, that's a problem for you to tackle in incubator, not here

Bolt is open to accepts commits from Gluten commits. Internally, every Bolt commit is checked against all DBMS integration including Spark-on-Gluten to ensure the stability. Currently Gluten backends-velox depends on ibm/velox which requires frequent rebases with upstream so Bolt backend reduces the long-term maintenance cost without messy rebase burden and providing first party supports on Gluten integration.

So, bolt is going to replace velox entirely in gluten, then the reduction of maintenance you've mentioned could actually happen. Do you have a clear plan for that?

As a fork of velox, why the integration for bolt result in a 260k+ patch? It doesn't look a right way to maintain. What's the amount of duplication? Consider a better approach such as new API exposure?

If spark adds a new feature or function, the gluten contributors need to apply an identical patch to velox and bolt, right? Otherwise, Technical debt accumulates quickly.

CI cost increase, the PR process as well

Assuming you refer to "vendor-neutral", Bolt project is in the process of submitting a proposal to ASF incubator, targeting 26Q1. We deeply value and encourage contributions from Gluten community to build the future of native engine acceleration. The merge of Bolt backend to the main branch is a foundational step to expose Spark-on-Bolt capability for community adoption which in return pushes the project to be "vendor-neutral".

A proposal to ASF incubator does not represent vendor-neutral.

You encourage others to contribute bolt, that's good, but bolt itself does not follow the upstream-first practice. We're in an awkward position.

Some comments of what I know:

So, bolt is going to replace velox entirely in gluten

It's up to customer actually, just like Clickhouse vs. Velox. Customer will do the choices depending on various reasons.

As a fork of velox, why the integration for bolt result in a 260k+ patch? It doesn't look a right way to maintain. What's the amount of duplication? Consider a better approach such as new API exposure?

Duplicated code is because we need to seperate the two backend's development initially. Then we may rebase the code, remove the duplications, align the APIs, achieve similar even better code layout compared to Velox vs. Clickhouse. It takes time and we need a start point.

Since I began testing Bolt

@FelixYBW would you share your test results for performance?

significant 22% performance enhancement over existing gluten backend on the TPC-DS 1T benchmark

22% e2e boost mentioned from Bolt vote mail, but I can not find any detailed test limitation like which data storage was tested? what major optimizations contributes this boost?

#11261 (comment)

SF500 on r6id.16xlarge, data is stored in local ssd. query is from https://github.com/prestodb/pbench/tree/tpc-queries-for-spark/benchmarks/tpc-ds/queries

See if you can reproduce.