-
Notifications
You must be signed in to change notification settings - Fork 241
Scripts are as follows:
spark-submit \ --deploy-mode cluster \ --master yarn \ --driver-memory 2g \ --num-executors 6 \ --executor-memory 4g \ --executor-cores 2 \ --queue eng \ --name pyspark_test \ --archives hdfs:///python_env/py_spark_env.zip#PY3 \ --jars "./blaze-engine-1.0-SNAPSHOT.jar" \ --conf spark.sql.extensions=org.apache.spark.sql.blaze.BlazeSparkSessionExtension \ --conf spark.executor.extraClassPath="./blaze-engine-1.0-SNAPSHOT.jar" \ --conf spark.yarn.appMasterEnv.PYSPARK_PYTHON=./PY3/py_spark_env/bin/python3.6 \ --conf spark.yarn.appMasterEnv.PYSPARK_DRIVER_PYTHON=./PY3/py_spark_env/bin/python3.6 \ --conf spark.default.parallelism=1000 \ testpyspark.py
All reactions
Replies: 1 comment 2 replies
@smartguo Thank you for your interest in Blaze. We apologize for any confusion you may be experiencing. We will be releasing a major version update soon that includes detailed user documentation.
All reactions
-
👍 2
2 replies
Thanks for your reply! Looking forward to your updates!
All reactions
can't wait for the updates!
All reactions
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment