-
Notifications
You must be signed in to change notification settings - Fork 618
-
Hi, I'm trying to read an iceberg table using Gluten (velox backend) and it's falling back, even though the table is not partitioned, and there are no deletions:
26/04/01 19:31:39 WARN GlutenFallbackReporter: Validation failed for plan: BatchScan spark-test.b7c164ad-9483-45ba-9235-8f60e6d94a19.EKKO_2y, due to:
Unsupported scan IcebergScan(table=spark-test.b7c164ad-9483-45ba-9235-8f60e6d94a19.EKKO_2y, branch=null, type=struct<1: MANDT: optional string, 2: EBELN: optional string, 3: BUKRS: optional string, 4: BSTYP: optional string, 5: BSART: optional string, 6: LOEKZ: optional string, 7: STATU: optional string, 8: AEDAT: optional date, 9: ERNAM: optional string, 10: LIFNR: optional string, 11: ZTERM: optional string, 12: ZBD1T: optional double, 13: ZBD2T: optional double, 14: ZBD3T: optional double, 15: ZBD1P: optional double, 16: ZBD2P: optional double, 17: EKORG: optional string, 18: EKGRP: optional string, 19: WAERS: optional string, 20: WKURS: optional double, 21: BEDAT: optional date, 22: KDATB: optional date, 23: KDATE: optional date, 24: BWBDT: optional date, 25: ANGDT: optional date, 26: BNDDT: optional date, 27: GWLDT: optional date, 28: IHRAN: optional date, 29: IHREZ: optional string, 30: VERKF: optional string, 31: LIFRE: optional string, 32: UNSEZ: optional string, 33: STCEG: optional string, 34: RELEASE_DATE: optional date, 35: ZZPOSOURCE: optional string, 36: ZZPOCLOSE: optional string, 37: ZZSHIPPOINT: optional string, 38: EQ_EINDT: optional date, 39: _CHANGE_DATE: optional timestamptz, 40: _rescued_data: optional string>, filters=[], runtimeFilters=], caseSensitive=false)
I'm confused because I'm running Spark 4 with the latest version of gluten-velox-bundle (jar downloaded from here https://dlcdn.apache.org/gluten/1.6.0/ from what I understand that is already built with the iceberg profile, please correct me if I'm wrong)
Can someone help me please figure out why I'm not able to enable the native scan?
Beta Was this translation helpful? Give feedback.
All reactions
@1b5d
In 1.6.0 release spark4.0 does not have iceberg support included, mostly due to not fully tested. Please have a try with below nightly 1.7 package:
https://nightlies.apache.org/gluten/nightly-release-jdk17-enhanced/
Replies: 1 comment 3 replies
-
@1b5d
In 1.6.0 release spark4.0 does not have iceberg support included, mostly due to not fully tested. Please have a try with below nightly 1.7 package:
https://nightlies.apache.org/gluten/nightly-release-jdk17-enhanced/
Beta Was this translation helpful? Give feedback.
All reactions
-
Thank you, the nightly package worked, but I guess we have to build our own release artifact if we want to use 1.6.0? I can only find snapshot versions
Beta Was this translation helpful? Give feedback.
All reactions
-
@zhouyuan a different quick question, do we know if credential vending is supported for iceberg tables?
Beta Was this translation helpful? Give feedback.
All reactions
-
@1b5d Hi, yeah you may need to build the Gluten package and enable iceberg module. Note that it's not fully tested on Spark-4.0 in the community as most users are still on Spark3.x
Here's the reference script for packaging:
https://github.com/apache/gluten/blob/main/dev/release/build-release.sh
The credential vending is not support now afaik, the native iceberg in Velox are now mostly focusing on I/O(read/write) - it should be the next stage goal as well as V3 support
Beta Was this translation helpful? Give feedback.