-
Notifications
You must be signed in to change notification settings - Fork 517
ORC-1200: Extracting encryption setup logic from WriterImpl - #1156
ORC-1200: Extracting encryption setup logic from WriterImpl #1156liujiawinds wants to merge 1 commit into
WriterImpl #1156Conversation
dongjoon-hyun
commented
Jun 11, 2022
Thank you for making a PR, @liujiawinds .
I set the milestone 1.9.0 which is the version of main branch.
For now, there is no plan of backporting this.
@dongjoon-hyun
dongjoon-hyun
left a comment
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
It's unclear to me why this is requested by Apache Flink community. For example,
- Do you have some documentation or reference in Apache Flink community?
- Is this Apache ORC design issue?
- Is this request aligned with other encryption handling in Apache Flink community (like Apache Parquet Modular Encryption handling)?
In general, branch-1.8 is already in feature freeze mode. I don't think this is required at v1.8.0 (currently).
Also, cc @omalley because he is the main author of Encryption Feature.
- Do you have some documentation or reference in Apache Flink community?
Yes, first we must create a stream based PhysicalFsWriter before WriterImpl initial. WriterImpl
Then, You can look into flink orc writer from OrcBulkWriterFactory.
Given the above, now I want to pass encryption settings to PhysicalFsWriter before WriterImpl initail, but the encryption setup at WriterImpl is so tightly coupled that the encryption variant cannot be obtained externally.
- Is this Apache ORC design issue?
Yes, I think it is.
- Is this request aligned with other encryption handling in Apache Flink community (like Apache Parquet Modular Encryption handling)?
Parquet has a similar class named FileEncryptionProperties. ParquetWriter
In general, branch-1.8 is already in feature freeze mode. I don't think this is required at v1.8.0 (currently).
It's ok to release in later versions. I will extract this part of code in flink.
dongjoon-hyun
commented
Jun 11, 2022
Thank you for the details.
liujiawinds
commented
Jun 13, 2022
What should I do next for this pr? I don't quite understand the meaning of it being marked as Changes requested state.
:)
dongjoon-hyun
commented
Jun 13, 2022
Instead of commenting, you are supposed to revise your PR description with your previous comment's content because only the PR title and description becomes a commit message.
What should I do next for this pr? I don't quite understand the meaning of it being marked as
Changes requestedstate.
:)
dongjoon-hyun
commented
Jun 13, 2022
After rethinking about this PR, I removed the milestone.
liujiawinds
commented
Jun 13, 2022
@dongjoon-hyun The PR description has been revised.
Uh oh!
There was an error while loading. Please reload this page.
What changes were proposed in this pull request?
Extracting the encryption setup logic as a tool class.
Why are the changes needed?
Because of flink's ORC writer is based of stream, we must create a stream based PhysicalFsWriter before WriterImpl initial. WriterImpl
Then, You can look into flink orc writer implementation from OrcBulkWriterFactory.
Given the above, now I want to pass encryption settings to PhysicalFsWriter before WriterImpl initail, but the encryption setup at WriterImpl is so tightly coupled that the encryption variant cannot be obtained externally.
How was this patch tested?
It doesn't introduce new features and passed all test cases.