[C++] read encrypted ORC file - #1993

Draft

zxf216 wants to merge 4 commits into

apache:main from

zxf216:read-encrypted-orc-file

Draft

[C++] read encrypted ORC file #1993
zxf216 wants to merge 4 commits into
apache:main from
zxf216:read-encrypted-orc-file

Conversation

@zxf216

@zxf216 zxf216 commented Aug 4, 2024 •

edited

Loading

Copy link

Copy Markdown

What changes were proposed in this pull request?

you can use C++ version code to read encrypted ORC files
ORC file structure:links

Why are the changes needed?

The C++ version is not yet able to read encrypted ORC files now

How was this patch tested?

orc::RowReaderOptions rowReaderOptions;
orc::ReaderOptions readerOpts;
std::shared_ptr<orc::InMemoryKeystore> keyStore = std::make_shared<orc::InMemoryKeystore>();
std::string passKey ="secret123";
keyStore->addKey("pii", 0, orc::EncryptionAlgorithm::AES_CTR_128,passKey);
readerOpts.setKeyProvider(keyStore);
printContents("/root/encryption.orc",readerOpts, rowReaderOptions);

@zxf216


 read encrypted ORC file

3328b37

@github-actions github-actions Bot added the CPP label

Aug 4, 2024

dongjoon-hyun

dongjoon-hyun reviewed

Aug 4, 2024

View reviewed changes

@dongjoon-hyun dongjoon-hyun left a comment

Copy link

Copy Markdown

Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you for making a PR, @zxf216 .

We have been tracking this via

#1183

BTW, could you make the CI happy by fixing all failures?

@wgtmac

wgtmac commented Aug 5, 2024

Copy link

Copy Markdown

Member

Thanks @zxf216! Let me take a first pass. cc @coderex2522 @ffacs

@zxf216

zxf216 commented Aug 6, 2024

Copy link

Copy Markdown

Author

Thank you for making a PR, @zxf216 .

We have been tracking this via

Add C++ API for Columnar Encryption #1183

BTW, could you make the CI happy by fixing all failures?
Okay

@dongjoon-hyun dongjoon-hyun modified the milestone: 2.1.0

Aug 11, 2024

@dongjoon-hyun

dongjoon-hyun marked this pull request as draft

August 11, 2024 18:32

@dongjoon-hyun

dongjoon-hyun commented Aug 11, 2024

Copy link

Copy Markdown

Member

I converted it to Draft because CIs fails still.

wgtmac

wgtmac requested changes

Aug 12, 2024

View reviewed changes

@wgtmac wgtmac left a comment

Copy link

Copy Markdown

Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks @zxf216 again for the excellent work! I just took a preliminary look and have left some inline comments. As you can see that we have various CI jobs for test cases, code format, license check, etc. I strongly recommend splitting the PR into smaller ones to make the review process easier.

c++/include/orc/Common.hh Outdated

c++/src/CMakeLists.txt

c++/src/TypeImpl.cc

}

return nullptr;

}

@wgtmac wgtmac Aug 12, 2024

Copy link

Copy Markdown

Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please revert the unnecessary change.

c++/include/orc/Reader.hh

struct ReaderOptionsPrivate;

struct RowReaderOptionsPrivate;

class KeyProvider;

@wgtmac wgtmac Aug 12, 2024

Copy link

Copy Markdown

Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The c++/include/orc folder is public and files in it will be installed. If the KeyProvider is also public, please move its definition to the include folder as well. Otherwise, we should remove the related function from here.

c++/src/security/InMemoryKeystore.cc

@@ -0,0 +1,247 @@

@wgtmac wgtmac Aug 12, 2024

Copy link

Copy Markdown

Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please note that we have a license check which does not allow custom modification to the license header in this repo. I'm not sure if this violates any ASF rule to add something like this. @dongjoon-hyun Do you have any concern?

c++/src/security/ReaderEncryption.hh

std::shared_ptr<SecretKeySpec> getStripeKey(long stripe);

};

// 加密变体,每个列一个

@wgtmac wgtmac Aug 12, 2024

Copy link

Copy Markdown

Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please use English for comment.

zxf216 and others added 3 commits

August 14, 2024 07:04

@zxf216 @wgtmac


 Update c++/include/orc/Common.hh

5ecbd41

Co-authored-by: Gang Wu <ustcwg@gmail.com>

@zxf216 @wgtmac


 Update c++/include/orc/Common.hh

5dc3331

Co-authored-by: Gang Wu <ustcwg@gmail.com>

@zxf216 @wgtmac


 Update c++/include/orc/Common.hh

f8201b7

Co-authored-by: Gang Wu <ustcwg@gmail.com>

@dongjoon-hyun dongjoon-hyun changed the title ~~(削除) read encrypted ORC file (削除ここまで)~~ (追記) [C++] read encrypted ORC file (追記ここまで)

Sep 20, 2024

@dongjoon-hyun

dongjoon-hyun commented Nov 4, 2024

Copy link

Copy Markdown

Member

Gentle ping, @zxf216 .

Labels

CPP

3 participants

@zxf216 @wgtmac @dongjoon-hyun

Uh oh!

Conversation

@zxf216 zxf216 commented Aug 4, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What changes were proposed in this pull request?

Why are the changes needed?

How was this patch tested?

Uh oh!

@dongjoon-hyun dongjoon-hyun left a comment

Choose a reason for hiding this comment

Uh oh!

wgtmac commented Aug 5, 2024

Uh oh!

zxf216 commented Aug 6, 2024

Uh oh!

dongjoon-hyun commented Aug 11, 2024

Uh oh!

@wgtmac wgtmac left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

@wgtmac wgtmac Aug 12, 2024

Choose a reason for hiding this comment

Uh oh!

@wgtmac wgtmac Aug 12, 2024

Choose a reason for hiding this comment

Uh oh!

@wgtmac wgtmac Aug 12, 2024

Choose a reason for hiding this comment

Uh oh!

@wgtmac wgtmac Aug 12, 2024

Choose a reason for hiding this comment

Uh oh!

dongjoon-hyun commented Nov 4, 2024

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

@zxf216 zxf216 commented Aug 4, 2024 •

edited

Loading