Skip to content

Navigation Menu

Sign in
Appearance settings

Search code, repositories, users, issues, pull requests...

Provide feedback

We read every piece of feedback, and take your input very seriously.

Saved searches

Use saved searches to filter your results more quickly
Sign up
Appearance settings

CRTHttpEngine IMDS timing out when attempting to establish a connect to Dynamo #1721

Open
Labels
bugThis issue is a bug. needs-triageThis issue or PR still needs to be triaged.
@cavanhussein

Description

Describe the bug

Hello once again,

Back in August I created an issue around how our micro service was failing to establish a connection to DynamoDB when using the CRTHttpEngine. In that issue a change was made to the ImdsCredentialsProvider in the hopes of addressing this issue. Unfortunately, I forgot that our subsequent production deploy switched our DynamoDBClient to use theOKHttpEngine instead of the CRTHttpEngine which led me to think our issue was resolved. Unfortunately, we are still seeing the IMDS error when using the CRTHttpEngine. Sadly, we are not able to use the OKHttpEngine since we are also seeing timeouts/stability issues as we continue adding more traffic to our instances. One important note is we've only really seen these IMDS errors in AP Northeast 2.

Below you can find our Bean configuration for our DynamoDBClient which is similar to our previous configuration, but with the ability to switch between httpClients:

@Configuration
@EnableConfigurationProperties(DynamoDBConfig::class)
open class DynamoDBConfiguration {
 @Bean(destroyMethod = "close")
 open fun dynamoDbClient(
 dynamoDBConfig: DynamoDBConfig
 ): DynamoDbClient {
 return DynamoDbClient {
 region = dynamoDBConfig.region
 if (dynamoDBConfig.kotlinHttpEngine == "CrtHttpEngine") {
 httpClient(CrtHttpEngine) {
 maxConcurrency = dynamoDBConfig.maxConcurrency.toUInt()
 }
 } else if (dynamoDBConfig.kotlinHttpEngine == "OkHttpEngine") {
 httpClient(OkHttpEngine) {
 maxConcurrency = dynamoDBConfig.maxConcurrency.toUInt()
 }
 } else {
 throw IllegalArgumentException("Unsupported dynamodb.kotlinHttpEngine type: '${dynamoDBConfig.kotlinHttpEngine}'")
 }
 if (!dynamoDBConfig.endpointOverride.isNullOrBlank()) {
 endpointUrl = Url.parse(dynamoDBConfig.endpointOverride!!)
 }
 }
 }
}

The DBHealthCheck Bean configuration we use to test our connection to DynamoDB can be seen here:

@Component
class DbHealthCheck(
 private val dynamoDbClient: DynamoDbClient,
 private val dynamoDBConfig: DynamoDBConfig,
 private val featureFlagService: FeatureFlagService,
 private val metadataCacheDispatcher: CoroutineDispatcher,
) : AbstractHealthIndicator() {
 private val logger = KotlinLogging.logger {}
 override fun doHealthCheck(builder: Health.Builder) {
 when {
 !featureFlagService.hasFeature(FeatureKey.CHECK_DB_CONNECTION) -> {
 logger.info { "health check disabled" }
 }
 runBlocking(metadataCacheDispatcher) {
 dynamoDbClient.describeTable(DescribeTableRequest {
 tableName =
 dynamoDBConfig.tables!!.metadataCache // nullability ignored intentionally since we want this to fail miserably if tables are null
 })
 }.table?.tableName != dynamoDBConfig.tables!!.metadataCache -> {
 logger.error { "db connection down" }
 builder.down().withDetails(mapOf("db" to mapOf(dynamoDBConfig.tables!!.metadataCache to "DOWN")))
 }
 else -> {
 logger.info { "db connection up" }
 builder.up().withDetails(mapOf("db" to mapOf(dynamoDBConfig.tables!!.metadataCache to "UP")))
 }
 }
 }
}

Here are a fresh set of logs when we see the IMDS errors:

{"timestamp":"2025-10-23 21:19:23.586","logger":"aws.smithy.kotlin.runtime.http.operation.OperationHandler","message":"operation started","context":"default"}
{"timestamp":"2025-10-23 21:19:23.612","logger":"aws.smithy.kotlin.runtime.auth.awscredentials.CachedCredentialsProvider","message":"refreshing credentials cache","context":"default"}
{"timestamp":"2025-10-23 21:19:23.614","logger":"aws.smithy.kotlin.runtime.identity.IdentityProviderChain","message":"Attempting to resolve identity from SystemPropertyCredentialsProvider","context":"default"}
{"timestamp":"2025-10-23 21:19:23.618","logger":"aws.sdk.kotlin.runtime.auth.credentials.SystemPropertyCredentialsProvider","message":"Attempting to load credentials from system properties aws.accessKeyId/aws.secretAccessKey/aws.sessionToken","context":"default"}
{"timestamp":"2025-10-23 21:19:23.619","logger":"aws.smithy.kotlin.runtime.identity.IdentityProviderChain","message":"unable to resolve identity from SystemPropertyCredentialsProvider: Missing value for system property `aws.accessKeyId`","context":"default"}
{"timestamp":"2025-10-23 21:19:23.620","logger":"aws.smithy.kotlin.runtime.identity.IdentityProviderChain","message":"Attempting to resolve identity from EnvironmentCredentialsProvider","context":"default"}
{"timestamp":"2025-10-23 21:19:23.620","logger":"aws.sdk.kotlin.runtime.auth.credentials.EnvironmentCredentialsProvider","message":"Attempting to load credentials from env vars AWS_ACCESS_KEY_ID/AWS_SECRET_ACCESS_KEY/AWS_SESSION_TOKEN","context":"default"}
{"timestamp":"2025-10-23 21:19:23.620","logger":"aws.smithy.kotlin.runtime.identity.IdentityProviderChain","message":"unable to resolve identity from EnvironmentCredentialsProvider: Missing value for environment variable `AWS_ACCESS_KEY_ID`","context":"default"}
{"timestamp":"2025-10-23 21:19:23.621","logger":"aws.smithy.kotlin.runtime.identity.IdentityProviderChain","message":"Attempting to resolve identity from aws.sdk.kotlin.runtime.auth.credentials.StsWebIdentityProvider@2165fef1","context":"default"}
{"timestamp":"2025-10-23 21:19:23.622","logger":"aws.smithy.kotlin.runtime.identity.IdentityProviderChain","message":"unable to resolve identity from aws.sdk.kotlin.runtime.auth.credentials.StsWebIdentityProvider@2165fef1: Required field `roleArn` could not be automatically inferred for StsWebIdentityCredentialsProvider. Either explicitly pass a value, set the environment variable `AWS_ROLE_ARN`, or set the JVM system property `aws.roleArn`","context":"default"}
{"timestamp":"2025-10-23 21:19:23.622","logger":"aws.smithy.kotlin.runtime.identity.IdentityProviderChain","message":"Attempting to resolve identity from ProfileCredentialsProvider","context":"default"}
{"timestamp":"2025-10-23 21:19:23.648","logger":"aws.sdk.kotlin.runtime.auth.credentials.ProfileCredentialsProvider","message":"Loading credentials from profile `default`","context":"default"}
{"timestamp":"2025-10-23 21:19:23.649","logger":"aws.smithy.kotlin.runtime.identity.IdentityProviderChain","message":"unable to resolve identity from ProfileCredentialsProvider: could not find source profile default","context":"default"}
{"timestamp":"2025-10-23 21:19:23.649","logger":"aws.smithy.kotlin.runtime.identity.IdentityProviderChain","message":"Attempting to resolve identity from EcsCredentialsProvider","context":"default"}
{"timestamp":"2025-10-23 21:19:23.650","logger":"aws.smithy.kotlin.runtime.identity.IdentityProviderChain","message":"unable to resolve identity from EcsCredentialsProvider: Container credentials URI not set","context":"default"}
{"timestamp":"2025-10-23 21:19:23.650","logger":"aws.smithy.kotlin.runtime.identity.IdentityProviderChain","message":"Attempting to resolve identity from ImdsCredentialsProvider","context":"default"}
{"timestamp":"2025-10-23 21:19:53.703","logger":"aws.smithy.kotlin.runtime.identity.IdentityProviderChain","message":"unable to resolve identity from ImdsCredentialsProvider: failed to load instance profile","context":"default"}
{"timestamp":"2025-10-23 21:19:53.704","logger":"aws.smithy.kotlin.runtime.awsprotocol.ClockSkewInterceptor","message":"service did not return \"Date\" header, skipping skew calculation","context":"default"}
{"timestamp":"2025-10-23 21:19:53.705","logger":"aws.smithy.kotlin.runtime.http.middleware.RetryMiddleware","message":"request failed with non-retryable error","context":"default"}
{"timestamp":"2025-10-23 21:19:53.705","level":"TRACE","thread":"metadata-cache-dispatcher-5","mdc":{"rpc":"DynamoDB.DescribeTable","sdkInvocationId":"215cc540-ed13-4d9f-b3d3-1f4df2ca6779","dd.trace_id":"68fa9bdb00000000745f8884b502b2c2","dd.service":"oauth","dd.span_id":"7661583407903012921"},"logger":"aws.smithy.kotlin.runtime.http.operation.OperationHandler","message":"operation failed","context":"default","exception":"aws.smithy.kotlin.runtime.identity.IdentityProviderException: No identity could be resolved from the chain: SystemPropertyCredentialsProvider -> EnvironmentCredentialsProvider -> aws.sdk.kotlin.runtime.auth.credentials.StsWebIdentityProvider@2165fef1 -> ProfileCredentialsProvider -> EcsCredentialsProvider -> ImdsCredentialsProvider
 at aws.smithy.kotlin.runtime.identity.IdentityProviderChain$resolve2ドル$chainException1ドル.invoke(IdentityProviderChain.kt:37)
 at aws.smithy.kotlin.runtime.identity.IdentityProviderChain$resolve2ドル$chainException1ドル.invoke(IdentityProviderChain.kt:37)
 at kotlin.SynchronizedLazyImpl.getValue(LazyJVM.kt:86)
 at aws.smithy.kotlin.runtime.identity.IdentityProviderChain$resolve$suspendImpl$$inlined$withSpan$default1ドル.invokeSuspend(CoroutineContextTraceExt.kt:139)
 at aws.smithy.kotlin.runtime.identity.IdentityProviderChain$resolve$suspendImpl$$inlined$withSpan$default1ドル.invoke(CoroutineContextTraceExt.kt)
 at aws.smithy.kotlin.runtime.identity.IdentityProviderChain$resolve$suspendImpl$$inlined$withSpan$default1ドル.invoke(CoroutineContextTraceExt.kt)
 at kotlinx.coroutines.intrinsics.UndispatchedKt.startUndspatched(Undispatched.kt:66)
 at kotlinx.coroutines.intrinsics.UndispatchedKt.startUndispatchedOrReturn(Undispatched.kt:43)
 at kotlinx.coroutines.BuildersKt__Builders_commonKt.withContext(Builders.common.kt:165)
 at kotlinx.coroutines.BuildersKt.withContext(Unknown Source)
 at aws.smithy.kotlin.runtime.identity.IdentityProviderChain.resolve$suspendImpl(IdentityProviderChain.kt:94)
 at aws.smithy.kotlin.runtime.identity.IdentityProviderChain.resolve(IdentityProviderChain.kt)
 at aws.smithy.kotlin.runtime.auth.awscredentials.CredentialsProviderChain.resolve(CredentialsProviderChain.kt:23)
 at aws.smithy.kotlin.runtime.auth.awscredentials.CachedCredentialsProvider$resolve3ドル.invokeSuspend(CachedCredentialsProvider.kt:63)
 at aws.smithy.kotlin.runtime.auth.awscredentials.CachedCredentialsProvider$resolve3ドル.invoke(CachedCredentialsProvider.kt)
 at aws.smithy.kotlin.runtime.auth.awscredentials.CachedCredentialsProvider$resolve3ドル.invoke(CachedCredentialsProvider.kt)
 at aws.smithy.kotlin.runtime.util.CachedValue.getOrLoad(CachedValue.kt:80)
 at aws.smithy.kotlin.runtime.auth.awscredentials.CachedCredentialsProvider.resolve(CachedCredentialsProvider.kt:61)
 at aws.sdk.kotlin.runtime.auth.credentials.DefaultChainCredentialsProvider.resolve(DefaultChainCredentialsProvider.kt:74)
 at aws.sdk.kotlin.runtime.auth.credentials.internal.ManagedCredentialsProvider.resolve(ManagedCredentialsProvider.kt)
 at aws.smithy.kotlin.runtime.http.operation.AuthHandler.call(SdkOperationExecution.kt:292)
 at aws.smithy.kotlin.runtime.http.operation.AuthHandler.call(SdkOperationExecution.kt:268)
 at aws.sdk.kotlin.runtime.http.middleware.AwsRetryHeaderMiddleware.handle(AwsRetryHeaderMiddleware.kt:39)
 at aws.sdk.kotlin.runtime.http.middleware.AwsRetryHeaderMiddleware.handle(AwsRetryHeaderMiddleware.kt:26)
 at aws.smithy.kotlin.runtime.io.middleware.DecoratedHandler.call(Middleware.kt:47)
 at aws.smithy.kotlin.runtime.io.middleware.Phase.handle(Phase.kt:68)
 at aws.smithy.kotlin.runtime.io.middleware.DecoratedHandler.call(Middleware.kt:47)
 at aws.smithy.kotlin.runtime.http.middleware.RetryMiddleware.tryAttempt-BWLJW6A(RetryMiddleware.kt:88)
 at aws.smithy.kotlin.runtime.http.middleware.RetryMiddleware.access$tryAttempt-BWLJW6A(RetryMiddleware.kt:34)
 at aws.smithy.kotlin.runtime.http.middleware.RetryMiddleware$handle$result$outcome1ドル$invokeSuspend$$inlined$withSpan$default1ドル.invokeSuspend(CoroutineContextTraceExt.kt:142)
 at aws.smithy.kotlin.runtime.http.middleware.RetryMiddleware$handle$result$outcome1ドル$invokeSuspend$$inlined$withSpan$default1ドル.invoke(CoroutineContextTraceExt.kt)
 at aws.smithy.kotlin.runtime.http.middleware.RetryMiddleware$handle$result$outcome1ドル$invokeSuspend$$inlined$withSpan$default1ドル.invoke(CoroutineContextTraceExt.kt)
 at kotlinx.coroutines.intrinsics.UndispatchedKt.startUndspatched(Undispatched.kt:66)
 at kotlinx.coroutines.intrinsics.UndispatchedKt.startUndispatchedOrReturn(Undispatched.kt:43)
 at kotlinx.coroutines.BuildersKt__Builders_commonKt.withContext(Builders.common.kt:165)
 at kotlinx.coroutines.BuildersKt.withContext(Unknown Source)
 at aws.smithy.kotlin.runtime.http.middleware.RetryMiddleware$handle$result$outcome1ドル.invokeSuspend(RetryMiddleware.kt:146)
 at aws.smithy.kotlin.runtime.http.middleware.RetryMiddleware$handle$result$outcome1ドル.invoke(RetryMiddleware.kt)
 at aws.smithy.kotlin.runtime.http.middleware.RetryMiddleware$handle$result$outcome1ドル.invoke(RetryMiddleware.kt)
 at aws.smithy.kotlin.runtime.retries.StandardRetryStrategy.doTryLoop(StandardRetryStrategy.kt:64)
 at aws.smithy.kotlin.runtime.retries.StandardRetryStrategy.retry$suspendImpl(StandardRetryStrategy.kt:44)
 at aws.smithy.kotlin.runtime.retries.StandardRetryStrategy.retry(StandardRetryStrategy.kt)
 at aws.smithy.kotlin.runtime.http.middleware.RetryMiddleware.handle(RetryMiddleware.kt:49)
 at aws.smithy.kotlin.runtime.http.middleware.RetryMiddleware.handle(RetryMiddleware.kt:34)
 at aws.smithy.kotlin.runtime.io.middleware.DecoratedHandler.call(Middleware.kt:47)
 at aws.smithy.kotlin.runtime.http.operation.MutateHandler.call(SdkOperationExecution.kt:265)
 at aws.smithy.kotlin.runtime.http.operation.MutateHandler.call(SdkOperationExecution.kt:262)
 at aws.smithy.kotlin.runtime.io.middleware.ModifyRequestMiddleware.handle(ModifyRequest.kt:26)
 at aws.smithy.kotlin.runtime.io.middleware.DecoratedHandler.call(Middleware.kt:47)
 at aws.smithy.kotlin.runtime.io.middleware.ModifyRequestMiddleware.handle(ModifyRequest.kt:26)
 at aws.smithy.kotlin.runtime.io.middleware.DecoratedHandler.call(Middleware.kt:47)
 at aws.smithy.kotlin.runtime.io.middleware.ModifyRequestMiddleware.handle(ModifyRequest.kt:26)
 at aws.smithy.kotlin.runtime.io.middleware.DecoratedHandler.call(Middleware.kt:47)
 at aws.smithy.kotlin.runtime.io.middleware.Phase.handle(Phase.kt:68)
 at aws.smithy.kotlin.runtime.io.middleware.DecoratedHan...

We would really appreciate some additional help on this. We are also considering switching to using the AWS Java SDK if we can find a path forward with the AWS Kotlin SDK :(

I will also create an AWS support ticket to see if there's an issue specific to AP Northeast 2.

Regression Issue

  • Select this option if this issue appears to be a regression.

Expected behavior

Below are logs when we successfully establish a connection to Dynamo:

{"timestamp":"2025-10-24 20:00:08.383","level":"TRACE","thread":"nonce-dispatcher-1","mdc":{"rpc":"DynamoDB.PutItem","sdkInvocationId":"dfd754a0-64ff-447f-a727-cfa970fbf758","loggingId":"9ee354d1-0950-4700-91b6-9e33a71463ef","dd.trace_id":"68fbdac7000000003b16da4a24d5a9cf","dd.service":"oauth","dd.span_id":"1348251663304369420"},"logger":"aws.smithy.kotlin.runtime.http.operation.OperationHandler","message":"operation started","context":"default"}
{"timestamp":"2025-10-24 20:00:08.429","level":"TRACE","thread":"nonce-dispatcher-1","mdc":{"rpc":"DynamoDB.PutItem","sdkInvocationId":"dfd754a0-64ff-447f-a727-cfa970fbf758","loggingId":"9ee354d1-0950-4700-91b6-9e33a71463ef","dd.trace_id":"68fbdac7000000003b16da4a24d5a9cf","dd.service":"oauth","dd.span_id":"1348251663304369420"},"logger":"aws.smithy.kotlin.runtime.auth.awscredentials.CachedCredentialsProvider","message":"refreshing credentials cache","context":"default"}
{"timestamp":"2025-10-24 20:00:08.432","level":"TRACE","thread":"nonce-dispatcher-1","mdc":{"rpc":"DynamoDB.PutItem","sdkInvocationId":"dfd754a0-64ff-447f-a727-cfa970fbf758","loggingId":"9ee354d1-0950-4700-91b6-9e33a71463ef","dd.trace_id":"68fbdac7000000003b16da4a24d5a9cf","dd.service":"oauth","dd.span_id":"1348251663304369420"},"logger":"aws.smithy.kotlin.runtime.identity.IdentityProviderChain","message":"Attempting to resolve identity from SystemPropertyCredentialsProvider","context":"default"}
{"timestamp":"2025-10-24 20:00:08.435","level":"TRACE","thread":"nonce-dispatcher-1","mdc":{"rpc":"DynamoDB.PutItem","sdkInvocationId":"dfd754a0-64ff-447f-a727-cfa970fbf758","loggingId":"9ee354d1-0950-4700-91b6-9e33a71463ef","dd.trace_id":"68fbdac7000000003b16da4a24d5a9cf","dd.service":"oauth","dd.span_id":"1348251663304369420"},"logger":"aws.sdk.kotlin.runtime.auth.credentials.SystemPropertyCredentialsProvider","message":"Attempting to load credentials from system properties aws.accessKeyId/aws.secretAccessKey/aws.sessionToken","context":"default"}
{"timestamp":"2025-10-24 20:00:08.436","level":"DEBUG","thread":"nonce-dispatcher-1","mdc":{"rpc":"DynamoDB.PutItem","sdkInvocationId":"dfd754a0-64ff-447f-a727-cfa970fbf758","loggingId":"9ee354d1-0950-4700-91b6-9e33a71463ef","dd.trace_id":"68fbdac7000000003b16da4a24d5a9cf","dd.service":"oauth","dd.span_id":"1348251663304369420"},"logger":"aws.smithy.kotlin.runtime.identity.IdentityProviderChain","message":"unable to resolve identity from SystemPropertyCredentialsProvider: Missing value for system property `aws.accessKeyId`","context":"default"}
{"timestamp":"2025-10-24 20:00:08.437","level":"TRACE","thread":"nonce-dispatcher-1","mdc":{"rpc":"DynamoDB.PutItem","sdkInvocationId":"dfd754a0-64ff-447f-a727-cfa970fbf758","loggingId":"9ee354d1-0950-4700-91b6-9e33a71463ef","dd.trace_id":"68fbdac7000000003b16da4a24d5a9cf","dd.service":"oauth","dd.span_id":"1348251663304369420"},"logger":"aws.smithy.kotlin.runtime.identity.IdentityProviderChain","message":"Attempting to resolve identity from EnvironmentCredentialsProvider","context":"default"}
{"timestamp":"2025-10-24 20:00:08.437","level":"TRACE","thread":"nonce-dispatcher-1","mdc":{"rpc":"DynamoDB.PutItem","sdkInvocationId":"dfd754a0-64ff-447f-a727-cfa970fbf758","loggingId":"9ee354d1-0950-4700-91b6-9e33a71463ef","dd.trace_id":"68fbdac7000000003b16da4a24d5a9cf","dd.service":"oauth","dd.span_id":"1348251663304369420"},"logger":"aws.sdk.kotlin.runtime.auth.credentials.EnvironmentCredentialsProvider","message":"Attempting to load credentials from env vars AWS_ACCESS_KEY_ID/AWS_SECRET_ACCESS_KEY/AWS_SESSION_TOKEN","context":"default"}
{"timestamp":"2025-10-24 20:00:08.438","level":"DEBUG","thread":"nonce-dispatcher-1","mdc":{"rpc":"DynamoDB.PutItem","sdkInvocationId":"dfd754a0-64ff-447f-a727-cfa970fbf758","loggingId":"9ee354d1-0950-4700-91b6-9e33a71463ef","dd.trace_id":"68fbdac7000000003b16da4a24d5a9cf","dd.service":"oauth","dd.span_id":"1348251663304369420"},"logger":"aws.smithy.kotlin.runtime.identity.IdentityProviderChain","message":"unable to resolve identity from EnvironmentCredentialsProvider: Missing value for environment variable `AWS_ACCESS_KEY_ID`","context":"default"}
{"timestamp":"2025-10-24 20:00:08.438","level":"TRACE","thread":"nonce-dispatcher-1","mdc":{"rpc":"DynamoDB.PutItem","sdkInvocationId":"dfd754a0-64ff-447f-a727-cfa970fbf758","loggingId":"9ee354d1-0950-4700-91b6-9e33a71463ef","dd.trace_id":"68fbdac7000000003b16da4a24d5a9cf","dd.service":"oauth","dd.span_id":"1348251663304369420"},"logger":"aws.smithy.kotlin.runtime.identity.IdentityProviderChain","message":"Attempting to resolve identity from aws.sdk.kotlin.runtime.auth.credentials.StsWebIdentityProvider@1404234","context":"default"}
{"timestamp":"2025-10-24 20:00:08.439","level":"DEBUG","thread":"nonce-dispatcher-1","mdc":{"rpc":"DynamoDB.PutItem","sdkInvocationId":"dfd754a0-64ff-447f-a727-cfa970fbf758","loggingId":"9ee354d1-0950-4700-91b6-9e33a71463ef","dd.trace_id":"68fbdac7000000003b16da4a24d5a9cf","dd.service":"oauth","dd.span_id":"1348251663304369420"},"logger":"aws.smithy.kotlin.runtime.identity.IdentityProviderChain","message":"unable to resolve identity from aws.sdk.kotlin.runtime.auth.credentials.StsWebIdentityProvider@1404234: Required field `roleArn` could not be automatically inferred for StsWebIdentityCredentialsProvider. Either explicitly pass a value, set the environment variable `AWS_ROLE_ARN`, or set the JVM system property `aws.roleArn`","context":"default"}
{"timestamp":"2025-10-24 20:00:08.439","level":"TRACE","thread":"nonce-dispatcher-1","mdc":{"rpc":"DynamoDB.PutItem","sdkInvocationId":"dfd754a0-64ff-447f-a727-cfa970fbf758","loggingId":"9ee354d1-0950-4700-91b6-9e33a71463ef","dd.trace_id":"68fbdac7000000003b16da4a24d5a9cf","dd.service":"oauth","dd.span_id":"1348251663304369420"},"logger":"aws.smithy.kotlin.runtime.identity.IdentityProviderChain","message":"Attempting to resolve identity from ProfileCredentialsProvider","context":"default"}
{"timestamp":"2025-10-24 20:00:08.461","level":"DEBUG","thread":"nonce-dispatcher-2","mdc":{"rpc":"DynamoDB.PutItem","sdkInvocationId":"dfd754a0-64ff-447f-a727-cfa970fbf758","loggingId":"9ee354d1-0950-4700-91b6-9e33a71463ef","dd.trace_id":"68fbdac7000000003b16da4a24d5a9cf","dd.service":"oauth","dd.span_id":"1348251663304369420"},"logger":"aws.sdk.kotlin.runtime.auth.credentials.ProfileCredentialsProvider","message":"Loading credentials from profile `default`","context":"default"}
{"timestamp":"2025-10-24 20:00:08.462","level":"DEBUG","thread":"nonce-dispatcher-2","mdc":{"rpc":"DynamoDB.PutItem","sdkInvocationId":"dfd754a0-64ff-447f-a727-cfa970fbf758","loggingId":"9ee354d1-0950-4700-91b6-9e33a71463ef","dd.trace_id":"68fbdac7000000003b16da4a24d5a9cf","dd.service":"oauth","dd.span_id":"1348251663304369420"},"logger":"aws.smithy.kotlin.runtime.identity.IdentityProviderChain","message":"unable to resolve identity from ProfileCredentialsProvider: could not find source profile default","context":"default"}
{"timestamp":"2025-10-24 20:00:08.462","level":"TRACE","thread":"nonce-dispatcher-2","mdc":{"rpc":"DynamoDB.PutItem","sdkInvocationId":"dfd754a0-64ff-447f-a727-cfa970fbf758","loggingId":"9ee354d1-0950-4700-91b6-9e33a71463ef","dd.trace_id":"68fbdac7000000003b16da4a24d5a9cf","dd.service":"oauth","dd.span_id":"1348251663304369420"},"logger":"aws.smithy.kotlin.runtime.identity.IdentityProviderChain","message":"Attempting to resolve identity from EcsCredentialsProvider","context":"default"}
{"timestamp":"2025-10-24 20:00:08.463","level":"DEBUG","thread":"nonce-dispatcher-2","mdc":{"rpc":"DynamoDB.PutItem","sdkInvocationId":"dfd754a0-64ff-447f-a727-cfa970fbf758","loggingId":"9ee354d1-0950-4700-91b6-9e33a71463ef","dd.trace_id":"68fbdac7000000003b16da4a24d5a9cf","dd.service":"oauth","dd.span_id":"1348251663304369420"},"logger":"aws.smithy.kotlin.runtime.identity.IdentityProviderChain","message":"unable to resolve identity from EcsCredentialsProvider: Container credentials URI not set","context":"default"}
{"timestamp":"2025-10-24 20:00:08.463","level":"TRACE","thread":"nonce-dispatcher-2","mdc":{"rpc":"DynamoDB.PutItem","sdkInvocationId":"dfd754a0-64ff-447f-a727-cfa970fbf758","loggingId":"9ee354d1-0950-4700-91b6-9e33a71463ef","dd.trace_id":"68fbdac7000000003b16da4a24d5a9cf","dd.service":"oauth","dd.span_id":"1348251663304369420"},"logger":"aws.smithy.kotlin.runtime.identity.IdentityProviderChain","message":"Attempting to resolve identity from ImdsCredentialsProvider","context":"default"}
{"timestamp":"2025-10-24 20:00:08.575","level":"DEBUG","thread":"nonce-dispatcher-6","mdc":{"rpc":"DynamoDB.PutItem","sdkInvocationId":"dfd754a0-64ff-447f-a727-cfa970fbf758","loggingId":"9ee354d1-0950-4700-91b6-9e33a71463ef","dd.trace_id":"68fbdac7000000003b16da4a24d5a9cf","dd.service":"oauth","dd.span_id":"1348251663304369420"},"logger":"aws.smithy.kotlin.runtime.http.operation.AuthHandler","message":"resolved endpoint: Endpoint(uri=https://dynamodb.ap-northeast-2.amazonaws.com, headers=null, attributes=aws.smithy.kotlin.runtime.collections.EmptyAttributes@30158934)","context":"default"}
{"timestamp":"2025-10-24 20:00:08.587","level":"TRACE","thread":"nonce-dispatcher-6","mdc":{"rpc":"DynamoDB.PutItem","sdkInvocationId":"dfd754a0-64ff-447f-a727-cfa970fbf758","loggingId":"9ee354d1-0950-4700-91b6-9e33a71463ef","dd.trace_id":"68fbdac7000000003b16da4a24d5a9cf","dd.service":"oauth","dd.span_id":"1348251663304369420"},"logger":"aws.smithy.kotlin.runtime.auth.awssigning.DefaultAwsSignerImpl","message":"String to sign:\nAWS4-HMAC-SHA256\n20251024T200008Z\n20251024/ap-northeast-2/dynamodb/aws4_request\n093bddb6bae66bdd730b5dbd1a6910cf47ea1fb2b3d675c78cb3c2aedf8029a7","context":"default"}
{"timestamp":"2025-10-24 20:00:08.588","level":"DEBUG","thread":"nonce-dispatcher-6","mdc":{"rpc":"DynamoDB.PutItem","sdkInvocationId":"dfd754a0-64ff-447f-a727-cfa970fbf758","loggingId":"9ee354d1-0950-4700-91b6-9e33a71463ef","dd.trace_id":"68fbdac7000000003b16da4a24d5a9cf","dd.service":"oauth","dd.span_id":"1348251663304369420"},"logger":"aws.smithy.kotlin.runtime.auth.awssigning.DefaultAwsSignerImpl","message":"Calculated signature: b29fcb6a39953b8c84ae3b823cb49fed5ca17bf8dcc3be389a745d2f83e073f1","context":"default"}
{"timestamp":"2025-10-24 20:00:08.640","level":"TRACE","thread":"DefaultDispatcher-worker-1","mdc":{"rpc":"DynamoDB.PutItem","sdkInvocationId":"dfd754a0-64ff-447f-a727-cfa970fbf758","dd.trace_id":"68fbdac7000000003b16da4a24d5a9cf","dd.service":"oauth","dd.span_id":"1348251663304369420"},"logger":"aws.smithy.kotlin.runtime.http.engine.crt.CrtHttpEngine","message":"Acquired connection 127768097831392","context":"default"}
{"timestamp":"2025-10-24 20:00:08.664","level":"TRACE","thread":"nonce-dispatcher-5","mdc":{"rpc":"DynamoDB.PutItem","sdkInvocationId":"dfd754a0-64ff-447f-a727-cfa970fbf758","loggingId":"9ee354d1-0950-4700-91b6-9e33a71463ef","dd.trace_id":"68fbdac7000000003b16da4a24d5a9cf","dd.service":"oauth","dd.span_id":"1348251663304369420"},"logger":"aws.smithy.kotlin.runtime.http.operation.OperationHandler","message":"operation completed successfully","context":"default"}
{"timestamp":"2025-10-24 20:00:08.664","level":"TRACE","thread":"nonce-dispatcher-5","mdc":{"rpc":"DynamoDB.PutItem","sdkInvocationId":"dfd754a0-64ff-447f-a727-cfa970fbf758","loggingId":"9ee354d1-0950-4700-91b6-9e33a71463ef","dd.trace_id":"68fbdac7000000003b16da4a24d5a9cf","dd.service":"oauth","dd.span_id":"1348251663304369420"},"logger":"aws.smithy.kotlin.runtime.http.engine.crt.CrtHttpEngine","message":"completing handler; cause=null","context":"default"}
{"timestamp":"2025-10-24 20:00:08.664","level":"TRACE","thread":"nonce-dispatcher-5","mdc":{"rpc":"DynamoDB.PutItem","sdkInvocationId":"dfd754a0-64ff-447f-a727-cfa970fbf758","loggingId":"9ee354d1-0950-4700-91b6-9e33a71463ef","dd.trace_id":"68fbdac7000000003b16da4a24d5a9cf","dd.service":"oauth","dd.span_id":"1348251663304369420"},"logger":"aws.smithy.kotlin.runtime.http.engine.crt.SdkStreamResponseHandler","message":"Closing connection 127768097831392","context":"default"}

Current behavior

See description.

Steps to Reproduce

We mentioned in the description, we are only really seeing this when we attempt to deploy our service to AP Northeast 2. In our testing we deploy an auto scaling group with 60 instances and find a handful that can never establish a connection to Dynamo.

As stated in our previous issue, if we restart our service using systemctl restart and the application can then successfully connect to Dynamo.

Possible Solution

No response

Context

No response

AWS SDK for Kotlin version

AWS Kotlin SDK (version 1.5.31), AWS CRT HTTP engine (version 1.5.14)

Platform (JVM/JS/Native)

JVM

Operating system and version

Ubuntu 24.04.1

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugThis issue is a bug. needs-triageThis issue or PR still needs to be triaged.

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions

        AltStyle によって変換されたページ (->オリジナル) /