-
Notifications
You must be signed in to change notification settings - Fork 241
i use auron in spark 。i submit spark job use : auron-spark-3.4_2.12-pre-osx-x86_64-6.0.0-incubating.jar to driver and executor config, but got an error:
25/10/31 07:47:08 INFO SecurityManager: SecurityManager: authentication enabled; ui acls disabled; users with view permissions: root; groups with view permissions: EMPTY; users with modify permissions: root; groups with modify permissions: EMPTY
25/10/31 07:47:08 INFO TransportClientFactory: Successfully created connection to spark-temp-id-b99b65c508-sql-b68ad69a393551c1-driver-svc.c2aas-prod.svc/10.244.1.26:7078 after 55 ms (36 ms spent in bootstraps)
Exception in thread "main" java.lang.reflect.UndeclaredThrowableException
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1894)
at org.apache.spark.deploy.SparkHadoopUtil.runAsSparkUser(SparkHadoopUtil.scala:62)
at org.apache.spark.executor.CoarseGrainedExecutorBackend$.run(CoarseGrainedExecutorBackend.scala:428)
at org.apache.spark.executor.CoarseGrainedExecutorBackend$.main(CoarseGrainedExecutorBackend.scala:417)
at org.apache.spark.executor.CoarseGrainedExecutorBackend.main(CoarseGrainedExecutorBackend.scala)
Caused by: java.lang.ClassNotFoundException: org.apache.spark.sql.execution.auron.shuffle.AuronShuffleManager
at java.base/jdk.internal.loader.BuiltinClassLoader.loadClass(BuiltinClassLoader.java:581)
at java.base/jdk.internal.loader.ClassLoaders$AppClassLoader.loadClass(ClassLoaders.java:178)
at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:527)
at java.base/java.lang.Class.forName0(Native Method)
at java.base/java.lang.Class.forName(Class.java:398)
at org.apache.spark.util.Utils$.classForName(Utils.scala:225)
at org.apache.spark.util.Utils$.instantiateSerializerOrShuffleManager(Utils.scala:2717)
at org.apache.spark.SparkEnv$.create(SparkEnv.scala:323)
at org.apache.spark.SparkEnv$.createExecutorEnv(SparkEnv.scala:212)
at org.apache.spark.executor.CoarseGrainedExecutorBackend$.$anonfun$run7ドル(CoarseGrainedExecutorBackend.scala:477)
at org.apache.spark.deploy.SparkHadoopUtil$$anon1ドル.run(SparkHadoopUtil.scala:63)
at org.apache.spark.deploy.SparkHadoopUtil$$anon1ドル.run(SparkHadoopUtil.scala:62)
at java.base/java.security.AccessController.doPrivileged(Native Method)
at java.base/javax.security.auth.Subject.doAs(Subject.java:423)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1878)
... 4 more
how to fix it , thank you。
All reactions
Replies: 2 comments 1 reply
It looks like you’re using a pre-release (pre) build, which doesn’t include all the required classes ?
Maybe you can try building the release version instead — for example :
./auron-build.sh --clean true --skiptests true --release --sparkver 3.4 --scalaver 2.12
Additionally, have you already placed the Auron JAR package in the Spark jars path?
You can find the detailed usage instructions at https://auron.apache.org/documents/getting-started.html#run-spark-job-with-auron-accelerator.
All reactions
thanks , I've solved this problem。
i have anthor question. How to verify that shuffle is using Auron?
All reactions
thanks , I've solved this problem。 i have anthor question. How to verify that shuffle is using Auron?
You can verify that the shuffle is using Auron in a few ways:
- Check the Spark SQL plan
When Auron is enabled, many shuffle-related nodes (e.g., ShuffleExchangeExec) will be replaced by native execution nodes such as:
- NativeShuffleExchange
- NativeHashAggregate
- NativeSortMergeJoin
- Look at the executor logs
During query execution, you’ll see log messages indicating native shuffle