Skip to content

Navigation Menu

Sign in
Appearance settings

Search code, repositories, users, issues, pull requests...

Provide feedback

We read every piece of feedback, and take your input very seriously.

Saved searches

Use saved searches to filter your results more quickly
Sign up
Appearance settings

Allow S3-Compatible Cloud Object Storage in CloudETL Example #1264

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
metadaddy wants to merge 4 commits into confluentinc:7.7.1-post
base: 7.7.1-post
Choose a base branch
Loading
from backblaze-b2-samples:allow-s3-compatible

Conversation

@metadaddy
Copy link

@metadaddy metadaddy commented Oct 8, 2024 •
edited
Loading

Description

This change allows the use of an S3-compatible cloud object store such as Backblaze B2 with the CloudETL example as an alternative to Amazon S3.

With this change, the user may configure an AWS profile with endpoint_url set to a value such as https://s3.us-west-004.backblazeb2.com in the config file. For example:

[profile b2-example]
endpoint_url = https://s3.us-west-004.backblazeb2.com
region = us-west-004

The setup_s3_storage.sh script reads this value (via aws configure get endpoint_url --profile $S3_PROFILE) into a new S3_ENDPOINT_URL environment variable which is used as store.url for the connectors. If the endpoint URL is not set in the profile, then the Amazon S3 global default, https://s3.amazonaws.com, is used.

There is also a fix to read-data.sh - the list-objects call was missing the --profile $S3_PROFILE, so it incorrectly used the default profile.

Author Validation

[x] cloud-etl

Reviewer Tasks

Describe the tasks/validation that the PR submitter is requesting to be done by the reviewer.

[ ] cloud-etl

@metadaddy metadaddy requested a review from a team as a code owner October 8, 2024 20:01
Copy link

confluent-cla-assistant bot commented Oct 8, 2024 •
edited
Loading

🎉 All Contributor License Agreements have been signed. Ready to merge.
✅ metadaddy
Please push an empty commit if you would like to re-run the checks to verify CLA status for all contributors.

Copy link

cla-assistant bot commented Oct 8, 2024 •
edited
Loading

CLA assistant check
All committers have signed the CLA.

Copy link

cla-assistant bot commented Oct 8, 2024

CLA assistant check
Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.
You have signed the CLA already but the status is still pending? Let us recheck it.

@metadaddy metadaddy changed the title (削除) Allow S3-Compatible Cloud Object Storage (削除ここまで) (追記) Allow S3-Compatible Cloud Object Storage in CloudETL Example (追記ここまで) Oct 8, 2024
Copy link
Author

Hi @davetroiano - you replied to me on #1262 last month, pointing to the PR for the CLI 4 compatibles samples. Any interest in taking this on? It shows how you can send data to any S3-compatible cloud object store (Backblaze, Minio, IBM Cloud Object Storage, etc etc) by setting the endpoint and region in the AWS profile (which you've likely already done if you're using any of the AWS SDKs/CLI with one of those providers).

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

1 participant

AltStyle によって変換されたページ (->オリジナル) /