Skip to content

Navigation Menu

Sign in
Appearance settings

Search code, repositories, users, issues, pull requests...

Provide feedback

We read every piece of feedback, and take your input very seriously.

Saved searches

Use saved searches to filter your results more quickly
Sign up
Appearance settings

Minimal Api with AddAWSLambdaHosting(LambdaEventSource.RestApi) - .Net 7 & with al2.provided custom runtime timeouts #1424

Unanswered
Simonl9l asked this question in Q&A
Discussion options

As part of a Stripe/Checkout integration with a request with a application/x-www-form-urlencoded payload, it requires that a 303 status is returned with the response header being the Stripe checkout url provided but he service:

 public static async Task<IResult> CreateCheckoutSession(
 HttpContext context, 
 [FromServices] StripeClientService stripeService
 )
 {
// setup options based on form values retrieved by context.Request.Form.TryGetValue("value", out var value);
 ...
 var session = await stripeService.CheckoutSessionService.CreateAsync(options);
 context.Response.Headers.Add("Location", session.Url);
 return Results.StatusCode(303);
}

This is run though both cloud front and an RestApi Gateway

here is the Lambda log (I extended the timeout to 30 secs):

2023年01月28日T15:48:59.695-08:00	info: Microsoft.AspNetCore.Http.Result.StatusCodeResult[1]
2023年01月28日T15:48:59.695-08:00	Setting HTTP status code 303.
2023年01月28日T15:48:59.696-08:00	2023年01月28日T23:48:59.696Z 3adb8e04-3983-4493-9e20-6e6ef04d7c23 [Information] Microsoft.AspNetCore.Http.Result.StatusCodeResult: Setting HTTP status code 303.
2023年01月28日T15:48:59.696-08:00	info: Microsoft.AspNetCore.Routing.EndpointMiddleware[1]
2023年01月28日T15:48:59.696-08:00	Executed endpoint 'HTTP: POST <<my route>> => CreateCheckoutSession'
2023年01月28日T15:49:27.696-08:00	2023年01月28日T23:49:27.695Z 3adb8e04-3983-4493-9e20-6e6ef04d7c23 Task timed out after 30.07 seconds
2023年01月28日T15:49:27.696-08:00	END RequestId: 3adb8e04-3983-4493-9e20-6e6ef04d7c23
2023年01月28日T15:49:27.696-08:00	REPORT RequestId: 3adb8e04-3983-4493-9e20-6e6ef04d7c23 Duration: 30068.57 ms Billed Duration: 31256 ms Memory Size: 512 MB Max Memory Used: 244 MB Init Duration: 1186.89 ms

Here are the Gateway logs:

2023年01月28日T15:48:55.939-08:00	(19da9199-f8e7-4656-8b03-792d5f94d0c6) Sending request to https://lambda.us-west-2.amazonaws.com/2015-03-31/functions/arn:aws:lambda:us-west-2:XXXXXXXX:function:XXXX-<Lambda>/invocations
2023年01月28日T15:49:24.939-08:00	(19da9199-f8e7-4656-8b03-792d5f94d0c6) Execution failed due to a timeout error
2023年01月28日T15:49:24.939-08:00	(19da9199-f8e7-4656-8b03-792d5f94d0c6) Method completed with status: 504

I need the 303 to get back to the browser so to will the use the Response Location and perform a GET

This all work cleanly locally with Kestrel

I have no idea what is cause the timeout - it seems to be happening the Handler/Wrapper - thoughts?
You must be logged in to vote

Replies: 7 comments 1 reply

Comment options

@Simonl9l I tried reproducing this and my simple cases of having an Minimal API return a 303 had no issues. This is my simple repo code

var builder = WebApplication.CreateBuilder(args);
// Add AWS Lambda support. When application is run in Lambda Kestrel is swapped out as the web server with Amazon.Lambda.AspNetCoreServer. This
// package will act as the webserver translating request and responses between the Lambda event source and ASP.NET Core.
builder.Services.AddAWSLambdaHosting(LambdaEventSource.RestApi);
var app = builder.Build();
app.UseHttpsRedirection(); 
app.MapGet("/", () => "Welcome to running ASP.NET Core Minimal API on AWS Lambda");
app.MapPost("/redirect-test", (HttpContext context) =>
{
 context.Response.Headers.Add("Location", "https://aws.amazon.com/");
 return Results.StatusCode(303);
});
app.Run();

From the logs it looks like the Lambda code got stuck returning back from the ASP.NET Core part back to Amazon.Lambda.AspNetCoreServer code and the function time out from API Gateway's point of view. Is there any additional middleware components that could be causing the request to not return?
You must be logged in to vote
0 replies
Comment options

@normj - Thanks for jumping in on this.

Ha - I see you edited to remove the app.UseAuthorization(); vs email :-)

I've rigged the Lambda code to do exactly as you have in code as above - plugged into the rest of the setup for routes etc so as not to have to mess with the client side code - but am still getting the same timeout issue.

To confirm that you deployed the above in an AWS Lambda?

Of note, we're building via Docker to a Single Executable that we Zip and deploy via CDK - and are using the al2.v14 runtime

Here are the detailed logs:

1675144521148,"INIT_START Runtime Version: provided:al2.v14	Runtime Version ARN: arn:aws:lambda:us-west-2::runtime:39d113fd8679cc06df6f699110bd034590026b16cab8188e33671d31ad03141c
1675144522022,"START RequestId: f5415ee9-6c5d-456b-9892-217f0df56699 Version: $LATEST
1675144522434,"info: Microsoft.AspNetCore.Hosting.Diagnostics[1]
1675144522434,"Request starting POST https://XXXXXX.execute-api.us-west-2.amazonaws.com/prod/<mypath> application/x-www-form-urlencoded 2288
1675144522434,"2023-01-31T05:55:22.434Z	f5415ee9-6c5d-456b-9892-217f0df56699	[Information] Microsoft.AspNetCore.Hosting.Diagnostics: Request starting POST https://XXXXXXX.execute-api.us-west-2.amazonaws.com/prod/<mypath> application/x-www-form-urlencoded 2288 
1675144522616,"info: Microsoft.AspNetCore.Routing.EndpointMiddleware[0]
1675144522616,"Executing endpoint 'HTTP: POST <my path> => TestRedirect'
1675144522618,"2023-01-31T05:55:22.618Z	f5415ee9-6c5d-456b-9892-217f0df56699	[Information] Microsoft.AspNetCore.Routing.EndpointMiddleware: Executing endpoint 'HTTP: POST <my path> => TestRedirect' 
1675144522618,"2023-01-31T05:55:22.618Z	f5415ee9-6c5d-456b-9892-217f0df56699	info	In Function Dummy CreateCheckoutSession
1675144522655,"info: Microsoft.AspNetCore.Http.Result.StatusCodeResult[1]
1675144522655,"Setting HTTP status code 303.
1675144522656,"2023-01-31T05:55:22.656Z	f5415ee9-6c5d-456b-9892-217f0df56699	[Information] Microsoft.AspNetCore.Http.Result.StatusCodeResult: Setting HTTP status code 303. 
1675144522656,"info: Microsoft.AspNetCore.Routing.EndpointMiddleware[1]
1675144522656,"Executed endpoint 'HTTP: POST <my path> => TestRedirect'
1675144552072,"2023-01-31T05:55:52.072Z f5415ee9-6c5d-456b-9892-217f0df56699 Task timed out after 30.05 seconds
1675144552072,"END RequestId: f5415ee9-6c5d-456b-9892-217f0df56699
1675144552072,"REPORT RequestId: f5415ee9-6c5d-456b-9892-217f0df56699	Duration: 30050.25 ms	Billed Duration: 30924 ms	Memory Size: 512 MB	Max Memory Used: 210 MB	Init Duration: 873.23 ms	
1675144552160,"INIT_START Runtime Version: provided:al2.v14	Runtime Version ARN: arn:aws:lambda:us-west-2::runtime:39d113fd8679cc06df6f699110bd034590026b16cab8188e33671d31ad03141c

I have retested locally and the 303 works as expected. So this is narrowing down the variables...

Should we have any concern with the use of the custom runtime?

We are running .Net 7 here - hence custom runtime!

Here is the dockerfile - were not AoT or Trimming with the latest finding:

FROM public.ecr.aws/amazonlinux/amazonlinux:2 AS base
WORKDIR /dotnet
RUN yum update -y && yum install -y clang llvm krb5-devel openssl-devel zip tar gzip compat-openssl10 libicu
RUN curl -sSL https://dot.net/v1/dotnet-install.sh | bash /dev/stdin -Channel 7.0 -InstallDir /usr/share/dotnet
RUN ln -s /usr/share/dotnet/dotnet /usr/bin/dotnet
FROM base AS build-env
WORKDIR /build
COPY . .
FROM build-env AS build-stage
ARG TARGET_PATH
ARG TARGET_RID
ARG AOT
ARG TRIM
ARG CONTAINED
ARG SINGLE
WORKDIR /build/$TARGET_PATH
RUN dotnet publish -r $TARGET_RID -c Release -o /build/publish --self-contained $CONTAINED -p:PublishSingleFile=$SINGLE -p:PublishAoT=$AOT -p:PublishTrimmed=$TRIM -p:StripSymbols=true
WORKDIR /build
RUN cp /build/publish/bootstrap bootstrap
RUN zip Lambda.zip bootstrap
FROM scratch AS export-stage
COPY --from=build-stage /build/Lambda.zip /
You must be logged in to vote
0 replies
Comment options

@normj did some digging with the team, and turned on LAMBDA_RUNTIMESUPPORT_DEBUG and got some interesting logging.

The behavior we are seeing is that cold runs succeed, but immediate subsequent runs (Warm) time out.

Tracking per the source code here

Per other notes will contact you offline.

Successful Cold Start:

2023年02月05日T12:59:29.723-08:00	START RequestId: cca7d7ae-6f5a-4f79-a391-a31dc856480f Version: $LATEST
2023年02月05日T12:59:29.760-08:00	2023年02月05日T20:59:29.759Z info [Info] Finished InternalClient.NextAsync
2023年02月05日T12:59:29.760-08:00	2023年02月05日T20:59:29.760Z cca7d7ae-6f5a-4f79-a391-a31dc856480f info [Info] Starting invoking handler
2023年02月05日T12:59:29.761-08:00	2023年02月05日T20:59:29.761Z cca7d7ae-6f5a-4f79-a391-a31dc856480f info Lambda Deserialize Amazon.Lambda.APIGatewayEvents.APIGatewayProxyRequest: {"resource":"/api/stripe/create-checkout-session","path":"/api/stripe/create-checkout-session","httpMethod":"POST","headers":{"Accept":"*/*","Accept-Encoding":"gzip, deflate, br","CloudFront-Forwarded-Proto":"https","CloudFront-Is-Desktop-Viewer":"true","CloudFront-Is-Mobile-Viewer":"false","CloudFront-Is-SmartTV-Viewer":"false","CloudFront-Is-Tablet-Viewer":"false","CloudFront-Viewer-ASN":"7018","CloudFront-Viewer-Country":"US","Host":"XXXXXXX.execute-api.us-west-2.amazonaws.com","User-Agent":"Thunder Client (https://www.thunderclient.com)","Via":"1.1 216f781ed7a0653429ac7a72888ca4c4.cloudfront.net (CloudFront)","X-Amz-Cf-Id":"8UzGIy6FycrLSPJ2AagvcxJ5W4Hm8NkREfWH6KfA0V1is9ujYEOKFQ==","X-Amzn-Trace-Id":"Root=1-63e018b1-340ae9ae1324904617c21ca2","X-Forwarded-For":"X.X.X.X, 70.132.18.80","X-Forwarded-Port":"443","X-Forwarded-Proto":"https"},"multiValueHeaders":{"Accept":["*/*"],"Accept-Encoding":["gzip, deflate, br"],"CloudFront-Forwarded-Proto":["https"],"CloudFront-Is-Desktop-Viewer":["true"],"CloudFront-Is-Mobile-Viewer":["false"],"CloudFront-Is-SmartTV-Viewer":["false"],"CloudFront-Is-Tablet-Viewer":["false"],"CloudFront-Viewer-ASN":["7018"],"CloudFront-Viewer-Country":["US"],"Host":["XXXXXXX.execute-api.us-west-2.amazonaws.com"],"User-Agent":["Thunder Client (https://www.thunderclient.com)"],"Via":["1.1 216f781ed7a0653429ac7a72888ca4c4.cloudfront.net (CloudFront)"],"X-Amz-Cf-Id":["8UzGIy6FycrLSPJ2AagvcxJ5W4Hm8NkREfWH6KfA0V1is9ujYEOKFQ=="],"X-Amzn-Trace-Id":["Root=1-63e018b1-340ae9ae1324904617c21ca2"],"X-Forwarded-For":["X.X.X.X, 70.132.18.80"],"X-Forwarded-Port":["443"],"X-Forwarded-Proto":["https"]},"queryStringParameters":null,"multiValueQueryStringParameters":null,"pathParameters":null,"stageVariables":null,"requestContext":{"resourceId":"zj7swi","resourcePath":"/api/stripe/create-checkout-session","httpMethod":"POST","extendedRequestId":"f4jLyEqbPHcFj1w=","requestTime":"05/Feb/2023:20:59:29 +0000","path":"/prod/api/stripe/create-checkout-session","accountId":"XXXXXXXXXXX","protocol":"HTTP/1.1","stage":"prod","domainPrefix":"XXXXXXX","requestTimeEpoch":1675630769694,"requestId":"65863ace-b75e-4998-a913-67197a45c584","identity":{"cognitoIdentityPoolId":null,"accountId":null,"cognitoIdentityId":null,"caller":null,"sourceIp":"X.X.X.X","principalOrgId":null,"accessKey":null,"cognitoAuthenticationType":null,"cognitoAuthenticationProvider":null,"userArn":null,"userAgent":"Thunder Client (https://www.thunderclient.com)","user":null},"domainName":"XXXXXXX.execute-api.us-west-2.amazonaws.com","apiId":"XXXXXXX"},"body":null,"isBase64Encoded":false}
2023年02月05日T12:59:30.034-08:00	dbug: Amazon.Lambda.AspNetCoreServer.AbstractAspNetCoreFunction[0]
2023年02月05日T12:59:30.034-08:00	ASP.NET Core Request PathBase: /prod, Path: /api/stripe/create-checkout-session
2023年02月05日T12:59:30.058-08:00	info: Microsoft.AspNetCore.Hosting.Diagnostics[1]
2023年02月05日T12:59:30.058-08:00	Request starting POST https://XXXXXXX.execute-api.us-west-2.amazonaws.com/prod/api/stripe/create-checkout-session - 0
2023年02月05日T12:59:30.061-08:00	dbug: Microsoft.AspNetCore.HostFiltering.HostFilteringMiddleware[0]
2023年02月05日T12:59:30.061-08:00	Wildcard detected, all requests with hosts will be allowed.
2023年02月05日T12:59:30.074-08:00	trce: Microsoft.AspNetCore.HostFiltering.HostFilteringMiddleware[2]
2023年02月05日T12:59:30.074-08:00	All hosts are allowed.
2023年02月05日T12:59:30.239-08:00	dbug: Microsoft.AspNetCore.Routing.Matching.DfaMatcher[1001]
2023年02月05日T12:59:30.239-08:00	1 candidate(s) found for the request path '/api/stripe/create-checkout-session'
2023年02月05日T12:59:30.255-08:00	dbug: Microsoft.AspNetCore.Routing.EndpointRoutingMiddleware[1]
2023年02月05日T12:59:30.255-08:00	Request matched endpoint 'HTTP: POST /api/stripe/create-checkout-session'
2023年02月05日T12:59:30.258-08:00	info: Microsoft.AspNetCore.Routing.EndpointMiddleware[0]
2023年02月05日T12:59:30.258-08:00	Executing endpoint 'HTTP: POST /api/stripe/create-checkout-session'
2023年02月05日T12:59:30.282-08:00	2023年02月05日T20:59:30.274Z cca7d7ae-6f5a-4f79-a391-a31dc856480f info In Function
2023年02月05日T12:59:30.282-08:00	info: Microsoft.AspNetCore.Http.Result.StatusCodeResult[1]
2023年02月05日T12:59:30.282-08:00	Setting HTTP status code 303.
2023年02月05日T12:59:30.282-08:00	info: Microsoft.AspNetCore.Routing.EndpointMiddleware[1]
2023年02月05日T12:59:30.282-08:00	Executed endpoint 'HTTP: POST /api/stripe/create-checkout-session'
2023年02月05日T12:59:30.322-08:00	dbug: Amazon.Lambda.AspNetCoreServer.AbstractAspNetCoreFunction[0]
2023年02月05日T12:59:30.322-08:00	Response Base 64 Encoded: False
2023年02月05日T12:59:30.334-08:00	info: Microsoft.AspNetCore.Hosting.Diagnostics[2]
2023年02月05日T12:59:30.334-08:00	Request finished POST https://XXXXXXX.execute-api.us-west-2.amazonaws.com/prod/api/stripe/create-checkout-session - 0 - 303 - - 265.9881ms
2023年02月05日T12:59:30.358-08:00	2023年02月05日T20:59:30.358Z cca7d7ae-6f5a-4f79-a391-a31dc856480f info Lambda Serialize Amazon.Lambda.APIGatewayEvents.APIGatewayProxyResponse: {"statusCode":303,"headers":{},"multiValueHeaders":{"Location":["https://aws.amazon.com/"],"Content-Type":[null]},"body":"","isBase64Encoded":false}
2023年02月05日T12:59:30.359-08:00	2023年02月05日T20:59:30.359Z cca7d7ae-6f5a-4f79-a391-a31dc856480f info [Info] Finished invoking handler
2023年02月05日T12:59:30.359-08:00	2023年02月05日T20:59:30.359Z cca7d7ae-6f5a-4f79-a391-a31dc856480f info [Info] Starting sending response
2023年02月05日T12:59:30.359-08:00	2023年02月05日T20:59:30.359Z cca7d7ae-6f5a-4f79-a391-a31dc856480f info [Info] Starting InternalClient.ResponseAsync
2023年02月05日T12:59:30.376-08:00	2023年02月05日T20:59:30.376Z cca7d7ae-6f5a-4f79-a391-a31dc856480f info [Info] Return from SendAsync
2023年02月05日T12:59:30.377-08:00	2023年02月05日T20:59:30.377Z cca7d7ae-6f5a-4f79-a391-a31dc856480f info [Info] Finished InternalClient.ResponseAsync
2023年02月05日T12:59:30.377-08:00	2023年02月05日T20:59:30.377Z cca7d7ae-6f5a-4f79-a391-a31dc856480f info [Info] Finished sending response
2023年02月05日T12:59:30.377-08:00	2023年02月05日T20:59:30.377Z cca7d7ae-6f5a-4f79-a391-a31dc856480f info [Info] Finished InvokeOnceAsync
2023年02月05日T12:59:30.377-08:00	2023年02月05日T20:59:30.377Z cca7d7ae-6f5a-4f79-a391-a31dc856480f info [Info] Starting InvokeOnceAsync
2023年02月05日T12:59:30.377-08:00	2023年02月05日T20:59:30.377Z cca7d7ae-6f5a-4f79-a391-a31dc856480f info [Info] Starting InternalClient.NextAsync
2023年02月05日T12:59:30.377-08:00	END RequestId: cca7d7ae-6f5a-4f79-a391-a31dc856480f
2023年02月05日T12:59:30.377-08:00	REPORT RequestId: cca7d7ae-6f5a-4f79-a391-a31dc856480f Duration: 654.25 ms Billed Duration: 655 ms Memory Size: 512 MB Max Memory Used: 184 MB

Subsequent call look like this:

2023年02月05日T12:59:32.578-08:00	START RequestId: 0c37b80f-9792-4d5c-a839-bd7174e5e7a2 Version: $LATEST
2023年02月05日T12:59:32.579-08:00	2023年02月05日T20:59:32.579Z cca7d7ae-6f5a-4f79-a391-a31dc856480f info [Info] Finished InternalClient.NextAsync
2023年02月05日T12:59:32.579-08:00	2023年02月05日T20:59:32.579Z 0c37b80f-9792-4d5c-a839-bd7174e5e7a2 info [Info] Starting invoking handler
2023年02月05日T12:59:32.579-08:00	2023年02月05日T20:59:32.579Z 0c37b80f-9792-4d5c-a839-bd7174e5e7a2 info Lambda Deserialize Amazon.Lambda.APIGatewayEvents.APIGatewayProxyRequest: {"resource":"/api/stripe/create-checkout-session","path":"/api/stripe/create-checkout-session","httpMethod":"POST","headers":{"Accept":"*/*","Accept-Encoding":"gzip, deflate, br","CloudFront-Forwarded-Proto":"https","CloudFront-Is-Desktop-Viewer":"true","CloudFront-Is-Mobile-Viewer":"false","CloudFront-Is-SmartTV-Viewer":"false","CloudFront-Is-Tablet-Viewer":"false","CloudFront-Viewer-ASN":"7018","CloudFront-Viewer-Country":"US","Host":"XXXXXXX.execute-api.us-west-2.amazonaws.com","User-Agent":"Thunder Client (https://www.thunderclient.com)","Via":"1.1 28663e5849ed20a9d037ca8066957990.cloudfront.net (CloudFront)","X-Amz-Cf-Id":"Y7lu82WopifSjhU9zqtyzI-fl0PpZ_-uJAivCk5oxBKHwfq9AO2tfQ==","X-Amzn-Trace-Id":"Root=1-63e018b4-676db62a56786bdb278c2512","X-Forwarded-For":"X.X.X.X, X.X.X.X","X-Forwarded-Port":"443","X-Forwarded-Proto":"https"},"multiValueHeaders":{"Accept":["*/*"],"Accept-Encoding":["gzip, deflate, br"],"CloudFront-Forwarded-Proto":["https"],"CloudFront-Is-Desktop-Viewer":["true"],"CloudFront-Is-Mobile-Viewer":["false"],"CloudFront-Is-SmartTV-Viewer":["false"],"CloudFront-Is-Tablet-Viewer":["false"],"CloudFront-Viewer-ASN":["7018"],"CloudFront-Viewer-Country":["US"],"Host":["XXXXXXX.execute-api.us-west-2.amazonaws.com"],"User-Agent":["Thunder Client (https://www.thunderclient.com)"],"Via":["1.1 28663e5849ed20a9d037ca8066957990.cloudfront.net (CloudFront)"],"X-Amz-Cf-Id":["Y7lu82WopifSjhU9zqtyzI-fl0PpZ_-uJAivCk5oxBKHwfq9AO2tfQ=="],"X-Amzn-Trace-Id":["Root=1-63e018b4-676db62a56786bdb278c2512"],"X-Forwarded-For":["X.X.X.X, X.X.X.X"],"X-Forwarded-Port":["443"],"X-Forwarded-Proto":["https"]},"queryStringParameters":null,"multiValueQueryStringParameters":null,"pathParameters":null,"stageVariables":null,"requestContext":{"resourceId":"zj7swi","resourcePath":"/api/stripe/create-checkout-session","httpMethod":"POST","extendedRequestId":"f4jMPEUfvHcF3bA=","requestTime":"05/Feb/2023:20:59:32 +0000","path":"/prod/api/stripe/create-checkout-session","accountId":"XXXXXXXXXXX","protocol":"HTTP/1.1","stage":"prod","domainPrefix":"XXXXXXX","requestTimeEpoch":1675630772562,"requestId":"f49b1012-890b-45d4-a3f7-e25d8d88581f","identity":{"cognitoIdentityPoolId":null,"accountId":null,"cognitoIdentityId":null,"caller":null,"sourceIp":"X.X.X.X","principalOrgId":null,"accessKey":null,"cognitoAuthenticationType":null,"cognitoAuthenticationProvider":null,"userArn":null,"userAgent":"Thunder Client (https://www.thunderclient.com)","user":null},"domainName":"XXXXXXX.execute-api.us-west-2.amazonaws.com","apiId":"XXXXXXX"},"body":null,"isBase64Encoded":false}
2023年02月05日T12:59:32.580-08:00	dbug: Amazon.Lambda.AspNetCoreServer.AbstractAspNetCoreFunction[0]
2023年02月05日T12:59:32.580-08:00	ASP.NET Core Request PathBase: /prod, Path: /api/stripe/create-checkout-session
2023年02月05日T12:59:32.580-08:00	info: Microsoft.AspNetCore.Hosting.Diagnostics[1]
2023年02月05日T12:59:32.580-08:00	Request starting POST https://XXXXXXX.execute-api.us-west-2.amazonaws.com/prod/api/stripe/create-checkout-session - 0
2023年02月05日T12:59:32.580-08:00	trce: Microsoft.AspNetCore.HostFiltering.HostFilteringMiddleware[2]
2023年02月05日T12:59:32.580-08:00	All hosts are allowed.
2023年02月05日T12:59:32.580-08:00	dbug: Microsoft.AspNetCore.Routing.Matching.DfaMatcher[1001]
2023年02月05日T12:59:32.580-08:00	1 candidate(s) found for the request path '/api/stripe/create-checkout-session'
2023年02月05日T12:59:32.580-08:00	dbug: Microsoft.AspNetCore.Routing.EndpointRoutingMiddleware[1]
2023年02月05日T12:59:32.580-08:00	Request matched endpoint 'HTTP: POST /api/stripe/create-checkout-session'
2023年02月05日T12:59:32.581-08:00	info: Microsoft.AspNetCore.Routing.EndpointMiddleware[0]
2023年02月05日T12:59:32.581-08:00	Executing endpoint 'HTTP: POST /api/stripe/create-checkout-session'
2023年02月05日T13:00:02.631-08:00	2023年02月05日T21:00:02.631Z 0c37b80f-9792-4d5c-a839-bd7174e5e7a2 Task timed out after 30.05 seconds
2023年02月05日T13:00:02.631-08:00	END RequestId: 0c37b80f-9792-4d5c-a839-bd7174e5e7a2
2023年02月05日T13:00:02.631-08:00	REPORT RequestId: 0c37b80f-9792-4d5c-a839-bd7174e5e7a2 Duration: 30053.53 ms Billed Duration: 30000 ms Memory Size: 512 MB Max Memory Used: 185 MB
2023年02月05日T13:00:02.735-08:00	INIT_START Runtime Version: provided:al2.v14 Runtime Version ARN: arn:aws:lambda:us-west-2::runtime:39d113fd8679cc06df6f699110bd034590026b16cab8188e33671d31ad03141c
2023年02月05日T13:00:04.065-08:00	dbug: Microsoft.Extensions.Hosting.Internal.Host[1]
2023年02月05日T13:00:04.065-08:00	Hosting starting
2023年02月05日T13:00:04.232-08:00	[Info] Using stdout and stderr for logging.
2023年02月05日T13:00:04.265-08:00	2023年02月05日T21:00:04.238Z info [Info] Starting InvokeOnceAsync
2023年02月05日T13:00:04.265-08:00	2023年02月05日T21:00:04.263Z info [Info] Starting InternalClient.NextAsync

Notes on a logging issues:

It seems that FinishedInternalClient.NextAsync is being called on the start of the second run:

2023年02月05日T12:59:32.579-08:00	2023年02月05日T20:59:32.579Z cca7d7ae-6f5a-4f79-a391-a31dc856480f info [Info] Finished InternalClient.NextAsync

we suspect these events should actually come out up top:

2023年02月05日T13:00:02.735-08:00	INIT_START Runtime Version: provided:al2.v14 Runtime Version ARN: arn:aws:lambda:us-west-2::runtime:39d113fd8679cc06df6f699110bd034590026b16cab8188e33671d31ad03141c
2023年02月05日T13:00:04.065-08:00	dbug: Microsoft.Extensions.Hosting.Internal.Host[1]
2023年02月05日T13:00:04.065-08:00	Hosting starting
2023年02月05日T13:00:04.232-08:00	[Info] Using stdout and stderr for logging.
2023年02月05日T13:00:04.265-08:00	2023年02月05日T21:00:04.238Z info [Info] Starting InvokeOnceAsync
2023年02月05日T13:00:04.265-08:00	2023年02月05日T21:00:04.263Z info [Info] Starting InternalClient.NextAsync
You must be logged in to vote
0 replies
Comment options

@normj - Just following up as we continue to research this.

We did find this possibly related issue: dotnet/aspnetcore#45373 and modified the handler as below (both adding a parameter and using the Delegate) - out assumption is that the Lambda Runtime Support is essentially wrapping the .Net Minimal API Middleware to serve up the request and wrangle the response in the the requisite event structure and that our 303 use case results in an empty response from a serialization back into the Lambda Event Response perspective and its intermittently stalling on the empty response?

ar builder = WebApplication.CreateBuilder(args);
// Add AWS Lambda support. When application is run in Lambda Kestrel is swapped out as the web server with Amazon.Lambda.AspNetCoreServer. This
// package will act as the webserver translating request and responses between the Lambda event source and ASP.NET Core.
builder.Services.AddAWSLambdaHosting(LambdaEventSource.RestApi);
var app = builder.Build();
app.UseHttpsRedirection();
app.MapGet("/", () => "Welcome to running ASP.NET Core Minimal API on AWS Lambda");
async Task<IResult> Handler(HttpContext context, CancellationToken ctx)
{
 Console.WriteLine("In Function");
 context.Response.Headers.Add("Location", "https://aws.amazon.com/");
 return Results.StatusCode(303);
}
Delegate delegated = Handler;
app.MapPost("/api/stripe/create-checkout-session", delegated);
app.Run();

Our observation are that we get mostly alternate succeed (with 303)/fail (with timeout) invocations, but do occasionally get a couple of succeed's in a row.

It essentially works every time after a START_INIT but not necessarily after a subsequent immediate invocation.

Of note if we switch the csproj back to <TargetFramework>net6.0</TargetFramework> and package/deploy with dotnet lambda tools using the. built in .Net 6 runtime it works. Thus to us is a clear indication that this is a runtime issue - do you agree?

However given our build chain and use of the Dockerfile above (comment) the dotnet publish is being set up as:

RUN dotnet publish -r linux-x64 \
 --configuration Release \
 --output /build/publish \
 --self-contained true \
 -p:PublishSingleFile=true \
 -p:PublishAoT=false \
 -p:PublishTrimmed=false \
 -p:StripSymbols=true \
 -p:PublishReadyToRun=true

and we are using public.ecr.aws/amazonlinux/amazonlinux:2 as out base, but do not that there is a Dockerfile in the repo (we believe is for the standard runtime - but referring to .Net 7) that is using public.ecr.aws/lambda/provided:al2 as its base.

Additionally our assumption is to install .Net 7 into the base container, and build from there so its compatible to the Amazon Linux libraries, but note that in multiple examples (per the .Net templates and the referenced Dockerfile above) the use of mcr.microsoft.com/dotnet/sdk:7.0-bullseye-slim as the builder environment is used. Is our assumption incorrect here, and should we use this or the alpine equivalent vs. what we are currently doing?

We also then emit a zip of the built lambda to the container host and deploy as a zip then in a container. Again do you see any issue with this approach - on basis that we configure the lambda as Custom Runtime an ensure the image is called bootstrap?

Finally I've tweaked the discussion subject as this is all .Net 7 runtime related

All help appreciated! Whilst short term we can probably workaround .Net 6 limitation vs .Net new features, we'd sure like to get a working/stable environment for .Net 7 and not have to wait for .net LTS support.
You must be logged in to vote
0 replies
Comment options

@normj just following up.

We have for the time being dropped back to the .net6 runtime for our Minimal API/LambdaHosting lambdas, until there is more support for .Net 7.

We do have observations that it would be good to share and have communicated directly as we don't fully understand the runtime code and don't want to commend based on that here, as noted other issues.

We're somewhat surprised others have not run into this issue with Minimal API with .Net 7 (with Custom runtime).
You must be logged in to vote
0 replies
Comment options

Catching up on the thread here, let me know if I missed any questions. I have taken your code here and published using .NET 7 as a single file to al2.provided. I admit I haven't tried building through the docker file yet. Just running the .NET CLI with the settings for singlefile. If you remove the docker build step and just run the .NET CLI does that give you a different behavior?

I find the description of how you are using containers to build and deploy a bit confusing. Any chance you could create me a sample repo I can clone and walk through?

Out of curiosity what are the Lambda memory settings you are using? Also what code size does Lambda report for the function. For me it reports 41MB.

  • FinishedInternalClient.NextAsync showing up in the subsequent invoke calls. That is correct behavior. When the runtime client makes the NextAsync call that basically tells Lambda nothing more is going on here and if there are no events to process go ahead and pause/freeze the compute environment. Then when the next event comes in the compute environment wakes up and starts processing the next event.

  • the response in the the requisite event structure and that our 303 use case results in an empty response from a serialization back into the Lambda Event Response perspective and its intermittently stalling on the empty response? I don't see how that would trigger the behavior you are seeing. Because even an empty response from Lambda would end the Lambda function. You are seeing the code for the Lambda function not returning the response back to the Lambda runtime client. You can confirm my understanding by setting the LAMBDA_NET_SERIALIZER_DEBUG environment variable to true and you should always see in the logs the JSON request and response for each invocation in the CloudWatch logs.

  • Of note if we switch the csproj back to <TargetFramework>net6.0</TargetFramework> and package/deploy with dotnet lambda tools using the. built in .Net 6 runtime it works. Thus to us is a clear indication that this is a runtime issue - do you agree? In this context what do you mean by runtime? The .NET runtime? The al2.provided runtime? or the Lambda runtime client? My guess would be the .NET runtime but I wish I could reproduce the issue to say anything for certain.

  • For your question about what base image to use. Unless you are using AOT it shouldn't matter.
You must be logged in to vote
0 replies
Comment options

@normj Apologies, things have been a tad crazy.

We had temporally reverted back to using .Net 6, as that had solved the issue, and temporarily has perhaps extended longer then planned; but now need to leverage some .Net 7 C#11 features...

Generally the timeout issue appears unrelated to the 303 use case.

We have updated to the latest amazon.lambda.tools tools to 5.7.2, to take advantage of the docker build process as defined for the .NET functions with native AOT compilation to produce a custom al2.provided image, leveraging the Amazon provided tools than our customer docker file.

Here is a link to a repo, this is based 100% from the standard serverless.AspNetCoreMinimalAPI template, but have modified it to incorporate aspects of the lambda.NativeAOT template as follows:

csproj

  • added I the <OutputType>, and <PublishAot> set to false
  • updated <TargetFramework> to net7.0
<PropertyGroup>
 <OutputType>Exe</OutputType>
 <TargetFramework>net7.0</TargetFramework>
 <ImplicitUsings>enable</ImplicitUsings>
 <Nullable>enable</Nullable>
 <GenerateRuntimeConfigurationFiles>true</GenerateRuntimeConfigurationFiles>
 <AWSProjectType>Lambda</AWSProjectType>
 <AssemblyName>bootstrap</AssemblyName>
 <!-- This property makes the build directory similar to a publish directory and helps the AWS .NET Lambda Mock Test Tool find project dependencies. -->
 <CopyLocalLockFileAssemblies>true</CopyLocalLockFileAssemblies>
 <!-- Generate Native AOT image during publishing to improve cold start time. -->
 <PublishAot>false</PublishAot>
 <!-- StripSymbols tells the compiler to strip debugging symbols from the final executable if we're on Linux and put them into their own file. 
 This will greatly reduce the final executable's size.-->
 <StripSymbols>true</StripSymbols>
 <!-- TrimMode partial will only trim assemblies marked as trimmable. To reduce package size make all assemblies trimmable and set TrimMode to full.
 If there are trim warnings during build, you can hit errors at runtime.-->
 <TrimMode>partial</TrimMode>
 <!-- Generate ready to run images during publishing to improvement cold starts. -->
 <PublishReadyToRun>true</PublishReadyToRun>
 <RestorePackagesWithLockFile>true</RestorePackagesWithLockFile>
 </PropertyGroup>

We also fixed the serverless.template, and aws-lambda-tools-defaults.json to indicate the al2.provided custom runtime.

The build process uses the:

  • dotnet lambda package tooling with the -ucfb True option to force the docker based build.
  • dotnet lambda deploy-serverless tooling wit he -pac <package as built above>

The repo has a GitHub Workflow that can be configured to build/deploy it as above, given the steps in the Readme.md

As such the repo can be cloned to reproduce the same outcomes.

Per the repo we initially added some addition Services for the .Net Code Generated Serialization, and a few other logging or AWS based services. This was on the basis it was related to additional code we were implementing than a base behavior.

We noted that all calls timed out!

From this, we then removed (commented out) all the additional code that differs from the serverless.AspNetCoreMinimalAPI in program.cs and redeployed. The reminder of the sample has remained untouched.

Our observations are that:

  • The / route will work consistently. However,
  • requests to the /Calculator/add/1/2 seems timeout every ~3-5 requests.

The Cloud Watch logging indicates the timedout requests never gets out of the middleware as the _logger.LogInformation($"{x} plus {y} is {x + y}"); never happens in these cases.

since we have the logging level set quite low:

{
 "Logging": {
 "LogLevel": {
 "Default": "Debug",
 "Microsoft.AspNetCore": "Debug"
 }
 }
}

the logging is quite verbose.

Our initial conclusion is that the default route works since there is NO DI or other request mapping; but in other scenarios the middleware is likely invoked.

Also we note that if we gap request by ~30 seconds it seem fine (for the same request). On rapid fire it chokes.

Our guess is that there is some kind of difference with how the AddAWSLambdaHosting and its middleware behavior works in Net7.0?

We do hope this is enough to prompt more investigation from you and the the AWS .Net team, as we're at a critical juncture to need to switch to .net 7.

Please advise!
Thanks!
You must be logged in to vote
1 reply
Comment options

related to logged Bug - #1543

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Category
Q&A
Labels
guidance Question that needs advice or information. response-requested Waiting on additional info and feedback. Will move to close soon in 7 days. module/lambda-client-lib
2 participants

AltStyle によって変換されたページ (->オリジナル) /