-
Notifications
You must be signed in to change notification settings - Fork 124
feat(io): refresh vended storage credentials before they expire - #892
Draft
plusplusjiajia wants to merge 3 commits into
Draft
feat(io): refresh vended storage credentials before they expire #892plusplusjiajia wants to merge 3 commits into
plusplusjiajia wants to merge 3 commits into
Conversation
plusplusjiajia
force-pushed
the
feat-vended-credential-refresh
branch
4 times, most recently
from
August 21, 2026 15:56
d92ef4d to
7088aed
Compare
wgtmac
commented
Aug 23, 2026
Member
Thanks for adding this feature! I think it is too large to review which may incur a long delay. Perhaps let's split it into smaller ones so we can review it one by one?
Member
Author
Thanks for adding this feature! I think it is too large to review which may incur a long delay. Perhaps let's split it into smaller ones so we can review it one by one?
@wgtmac Thanks — split into a stack, smallest first:
- test(io): stop probing the EC2 metadata service in S3 tests #897 — S3 tests stop probing the EC2 metadata service (54s → 0.3s).
- refactor(io): hand out S3 delegates by shared_ptr and build them off the lock #898 — ArrowS3FileIO gets refactor(io): make FileIO resolution registry-driven #889 's concurrency treatment. No behavior change.
- feat(rest): load table credentials from the LoadCredentials endpoint #899 (draft until refactor(io): hand out S3 delegates by shared_ptr and build them off the lock #898 merges) — LoadCredentials plumbing, not yet invoked.
- This PR shrinks to just the refresh policy.
Each builds and passes the full suite standalone; the diff here narrows as each one merges.
plusplusjiajia
marked this pull request as draft
August 24, 2026 09:56
plusplusjiajia
force-pushed
the
feat-vended-credential-refresh
branch
from
August 24, 2026 10:49
7088aed to
b06186d
Compare
plusplusjiajia
force-pushed
the
feat-vended-credential-refresh
branch
from
September 9, 2026 07:59
b06186d to
afe009b
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Vended storage credentials are short-lived, so a long scan or a reused table eventually fails on expired ones. This fills the
TODO(gangwu)left inarrow_s3_file_io.ccby #719.How
ArrowS3FileIOreadss3.session-token-expires-at-msand, five minutes before the earliest applied credential expires, calls aStorageCredentialRefresherand rebuilds its per-prefix delegates.RestCatalogsupplies that callback from theLoadCredentialsendpoint when the server advertises it;ResolvingFileIOpasses it down. Aligned with Java's
VendedCredentialsProvider: same lead time, same lazy refresh rather than a background thread, same refusal of an empty credential list.SupportsStorageCredentials::credentials()now returns by value: a refresh can replace the vector concurrently. All in-tree callers are tests.