Skip to content

Navigation Menu

Sign in
Sign up

Is PME format supported? #1759

Unanswered
Dormant7 asked this question in Q&A
Discussion options

I'm using Auron to process data stored in PME (Parquet Modular Encryption) format. I set decrypt config in source code .
image
image
image

but I meet an error.
image
I've configured it, but it's still not working — not sure what's wrong.

3q

You must be logged in to vote

Replies: 1 comment

Comment options

I don't think PME is supported on the native Parquet path yet — at least, that's what I'm seeing in the code as of 2026年07月03日:

  • The native scan seems to hardcode decryption off — file_decryption_properties: None in native-engine/datafusion-ext-plans/src/parquet_exec.rs.
  • I couldn't find a path that passes keys from Spark down to native — no key/crypto fields in the ParquetScanExecNode / FileScanExecConf proto, and nothing on the Scala side reading parquet.encryption.* or a KMS crypto.factory.
  • The write path looks like it explicitly rejects it — assert(!encryptEnabled, "not supported writing encrypted table") in NativeParquetInsertIntoHiveTableBase.scala.

As for why your patch might still error: the "encrypted footer but decryption properties were not provided" message seems to come from footer/metadata parsing, which runs before any column decryption. Auron overrides AsyncFileReader::get_metadata with its own reader (parquet_exec.rs:399), and that override takes ArrowReaderOptions as an unused _options argument and reads the footer via the deprecated fetch_parquet_metadata, which as far as I can tell has no decryption path — so the properties you set on the ParquetOpener may not be reaching the footer parser, even though the field is populated. I could be missing something, though. If you wanted to take this further, a GitHub issue might be a good place to scope it.

You must be logged in to vote
0 replies
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Category
Q&A
Labels
None yet

AltStyle によって変換されたページ (->オリジナル) /