-
Couldn't load subscription status.
- Fork 947
-
Hi! I have a CRT s3 client and transfer manager setup like this:
s3Client = S3AsyncClient.crtBuilder() .credentialsProvider( StaticCredentialsProvider.create( AwsBasicCredentials.create( s3Endpoint.accessKey, s3Endpoint.secretKey ) ) ) .maxConcurrency(s3Properties.maxConcurrent) .region(Region.US_EAST_1) .endpointOverride(s3Endpoint.url) .forcePathStyle(true) .checksumValidationEnabled(false) .build() transferManager = S3TransferManager.builder() .s3Client(s3Client) .build()
When running the code in Kubernetes and trying to upload a large file (80-100GB), the pod uses way more memory than it's been assigned to do (>9GB in a pod with 8GB limit), and subsequently gets OOM-killed; since the java heap is smaller than 1GB, I'm suspecting that the CRT libraries is the reason for the high memory consumption. The same behavior occurs when uploading the same file using the CLI.
Is there any way to restrict how much memory the CRT client is allowed to use? The closes I've seen is setting maxConcurrency, but it seems very roundabout.
Beta Was this translation helpful? Give feedback.
All reactions
Replies: 3 comments
-
After some thinking, this seems very much like unintended behavior. I've created an issue for it, see #4034
Beta Was this translation helpful? Give feedback.
All reactions
-
Hello! Reopening this discussion to make it searchable.
Beta Was this translation helpful? Give feedback.
All reactions
-
@Lunkers and others. This has been resolved with #6323 (comment)
Beta Was this translation helpful? Give feedback.