Skip to content

Navigation Menu

Sign in
Appearance settings

Search code, repositories, users, issues, pull requests...

Provide feedback

We read every piece of feedback, and take your input very seriously.

Saved searches

Use saved searches to filter your results more quickly

Sign up
Appearance settings

Restricting CRT s3 client memory usage? #4033

Unanswered
Lunkers asked this question in Q&A
Discussion options

Hi! I have a CRT s3 client and transfer manager setup like this:

s3Client = S3AsyncClient.crtBuilder()
 .credentialsProvider(
 StaticCredentialsProvider.create(
 AwsBasicCredentials.create(
 s3Endpoint.accessKey,
 s3Endpoint.secretKey
 )
 )
 )
 .maxConcurrency(s3Properties.maxConcurrent)
 .region(Region.US_EAST_1)
 .endpointOverride(s3Endpoint.url)
 .forcePathStyle(true)
 .checksumValidationEnabled(false)
 .build()
 
 transferManager = S3TransferManager.builder()
 .s3Client(s3Client)
 .build()

When running the code in Kubernetes and trying to upload a large file (80-100GB), the pod uses way more memory than it's been assigned to do (>9GB in a pod with 8GB limit), and subsequently gets OOM-killed; since the java heap is smaller than 1GB, I'm suspecting that the CRT libraries is the reason for the high memory consumption. The same behavior occurs when uploading the same file using the CLI.

Is there any way to restrict how much memory the CRT client is allowed to use? The closes I've seen is setting maxConcurrency, but it seems very roundabout.

You must be logged in to vote

Replies: 3 comments

Comment options

After some thinking, this seems very much like unintended behavior. I've created an issue for it, see #4034

You must be logged in to vote
0 replies
Comment options

Hello! Reopening this discussion to make it searchable.

You must be logged in to vote
0 replies
Comment options

@Lunkers and others. This has been resolved with #6323 (comment)

You must be logged in to vote
0 replies
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Category
Q&A
Labels
None yet

AltStyle によって変換されたページ (->オリジナル) /