-
Notifications
You must be signed in to change notification settings - Fork 241
what is the difference between blaze+ESS and blaze+Celeborn? #852
In my perception, celeborn is similar to ESS(external shuffle service), the difference is only that celeborn can move the shuffle service from NodeManager to external.
However, in our tpcds1000GB dataset test, we found that the query time of blaze+ESS is 20% less than the native spark, but the query time of blaze+celeborn is not significantly reduced than the native spark, and even has a small increase.
The following is the test environment:
- CDP cluster
- spark-3.1.1
- blaze-v4.0.1
- celeborn-v0.5.3
- NodeManager - 17 nodes
- celeborn - 8 nodes+SSD
The following are the key configurations:
- blaze+ESS
spark.blaze.enable true
spark.sql.extensions org.apache.spark.sql.blaze.BlazeSparkSessionExtension
spark.shuffle.manager org.apache.spark.sql.execution.blaze.shuffle.BlazeShuffleManager
spark.memory.offHeap.enabled false
- blaze+celeborn
spark.shuffle.manager org.apache.spark.sql.execution.blaze.shuffle.celeborn.BlazeCelebornShuffleManager
spark.celeborn.master.endpoints xxxxx:9097
spark.celeborn.client.spark.shuffle.writer hash
spark.celeborn.client.push.replicate.enabled false
spark.blaze.enable true
spark.sql.extensions org.apache.spark.sql.blaze.BlazeSparkSessionExtension
spark.memory.offHeap.enabled false
All reactions
Replies: 1 comment
Have you tested the following two setups?
- Vanilla Spark + ESS
- Vanilla Spark + Celeborn
I think the difference between ESS and Celeborn are that: For ESS spark writes shuffle data to local disk, for Celeborn spark writes shuffle data to remote server. So the performance is really determined by the disks IOPS/throughput(on ESS and Celeborn) and the network bandwidth.