Why is my TPC-DS SQL query running slower with Auron compared to native Spark? #1976

Thanks for reporting this. A couple of things might help narrow it down.

The config you shared looks close to the README quick-start, which is meant as a minimal starting point rather than a benchmarking profile. It might be worth comparing it against the setup behind the published 1TB numbers (https://auron.apache.org/documents/benchmarks.html) — especially spark.executor.memoryOverhead, spark.executor.cores, and spark.auron.memoryFraction, since native execution in Auron draws its memory from the overhead region rather than Spark's off-heap pool (offHeap.enabled=false is expected). Do the two configs differ much?

A few questions that would help:

1. Does the slowdown show up across most queries, or mainly a few like q6?
2. In EXPLAIN FORMATTED (or the Spark UI SQL tab), does the Auron plan mostly use Native*/Auron* operators, or are there stretches of regular Spark operators mixed in?
3. In the Spark UI for q6, how do the spill and shuffle read/write numbers compare between the Auron and native-Spark runs?

That should give us a good sense of whether it's a memory/spill effect or something in how the plan is being executed.