-
Notifications
You must be signed in to change notification settings - Fork 241
Describe the bug
Why do I only print Rust logs when I run code on Spark333 and execute SQL commands in Sparkshell? How do I print logs related to Scala?
To Reproduce
- generate jar package.
SHIM=spark333
MODE=release
mvn package -P"${SHIM}" -P"${MODE}"
- run spark-shell
/home/qcsun/wql_zx_files/spark333/bin/spark-shell --name run_test \
--master local[4] --deploy-mode client \
--jars /home/qcsun/wql_zx_files/blaze-engine/target/blaze-engine-spark333-release-2.0.9.1-SNAPSHOT.jar
3.execute some code
import org.apache.spark.sql.SparkSession
val spark = SparkSession.builder().appName("Spark SQL Example").getOrCreate()
spark.sql("CREATE DATABASE IF NOT EXISTS test_db")
spark.sql("USE test_db")
spark.sql("DROP TABLE IF EXISTS test_table")
spark.sql("""
CREATE TABLE IF NOT EXISTS test_table (
id INT,
value DOUBLE
)
USING PARQUET
""")
val data = spark.range(100000).map(id => (
id,
Math.random() * 100,
)).toDF("id", "value")
data.write.mode("overwrite").insertInto("test_table")
sql("SELECT AVG(value) FROM test_table").show()
- I found only rust log can be printed. So why?!
Screenshots
image
Desktop (please complete the following information):
- OS: [centos7]
- Version [spark333]
All reactions
Replies: 1 comment
Have you checked log levels and configurations?
I think you can try to set log4j.rootCategory=INFO,console in file log4j.properties
All reactions
0 replies
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment