-
Notifications
You must be signed in to change notification settings - Fork 618
Running Apache Gluten locally on macOS (Apple Silicon) with Spark 3.5 – libgluten.dylib missing #11535
-
Hi Gluten community,
I’m trying to run Apache Gluten locally on macOS (Apple Silicon / M1) with Apache Spark 3.5.3, and I’m running into a native library loading issue.
I wanted to check whether local macOS support is expected to work today, or if Gluten is currently Linux-only for the Velox backend.
OS: macOS (Apple Silicon – aarch64)
Spark: 3.5.3 (spark-3.5.3-bin-hadoop3)
Java: OpenJDK 17
Gluten:
gluten-velox-bundle-spark3.5_2.12-linux_amd64-1.5.0.jar
Backend: Velox
Execution mode: local
$SPARK_HOME/bin/spark-shell \
--jars $GLUTEN_JAR \
--conf spark.plugins=org.apache.gluten.GlutenPlugin \
--conf spark.gluten.sql.enable=true \
--conf spark.gluten.execution.backend=velox \
--conf spark.memory.offHeap.enabled=true \
--conf spark.memory.offHeap.size=2g \
--conf spark.driver.extraClassPath=$GLUTEN_JAR \
--conf spark.executor.extraClassPath=$GLUTEN_JAR \
--conf spark.sql.columnVector.offheap.enabled=true \
--conf spark.gluten.sql.fallback.enabled=true \
--conf spark.driver.bindAddress=127.0.0.1 \
--conf spark.driver.host=127.0.0.1
Error Observed
WARN Discovery: Found duplicated component class
org.apache.gluten.component.VeloxPaimonComponent in the classpath, ignoring.
ERROR SparkContext: Error initializing SparkContext.
org.apache.gluten.exception.GlutenException:
java.io.FileNotFoundException: darwin/aarch64/libgluten.dylib
at org.apache.gluten.jni.JniLibLoader.load(JniLibLoader.java:89)
Questions
Is running Gluten + Velox locally on macOS supported today?
Are there prebuilt macOS (darwin/aarch64) native binaries for Gluten?
I took jar from here
https://downloads-he-de-2.apache.org/incubator/gluten/1.4.0-incubating/apache-gluten-1.4.0-incubating-bin-spark35.tar.gz
Beta Was this translation helpful? Give feedback.
All reactions
Replies: 1 comment
-
Please see the document #11547
Beta Was this translation helpful? Give feedback.