-
Notifications
You must be signed in to change notification settings - Fork 241
Replies: 1 comment
I don't think PME is supported on the native Parquet path yet — at least, that's what I'm seeing in the code as of 2026年07月03日:
- The native scan seems to hardcode decryption off —
file_decryption_properties: Noneinnative-engine/datafusion-ext-plans/src/parquet_exec.rs. - I couldn't find a path that passes keys from Spark down to native — no key/crypto fields in the
ParquetScanExecNode/FileScanExecConfproto, and nothing on the Scala side readingparquet.encryption.*or a KMScrypto.factory. - The write path looks like it explicitly rejects it —
assert(!encryptEnabled, "not supported writing encrypted table")inNativeParquetInsertIntoHiveTableBase.scala.
As for why your patch might still error: the "encrypted footer but decryption properties were not provided" message seems to come from footer/metadata parsing, which runs before any column decryption. Auron overrides AsyncFileReader::get_metadata with its own reader (parquet_exec.rs:399), and that override takes ArrowReaderOptions as an unused _options argument and reads the footer via the deprecated fetch_parquet_metadata, which as far as I can tell has no decryption path — so the properties you set on the ParquetOpener may not be reaching the footer parser, even though the field is populated. I could be missing something, though. If you wanted to take this further, a GitHub issue might be a good place to scope it.