FastAPI: PDF upload works when called directly inside Kubernetes or locally, but results in blank files when routed through API Gateway/NGINX

Question

I’m running a FastAPI service in Kubernetes that accepts multipart/form-data uploads and stores files in S3. The upload endpoint works fine when I call it directly inside the cluster using the service name, but when I go through our external route (API Gateway → CloudFront → NGINX → ALB → K8s service), the upload returns 201 Created, yet the uploaded PDF in S3 is blank/corrupted.

What I tested

Direct internal call (works fine):

curl -v -X POST \
 -F "documentType=POD" \
 -F "organizationId=1005" \
 -F "accessType=test" \
 -F "file=@/tmp/test.pdf;type=application/pdf" \
 http://doc-service.default.svc.cluster.local:8500/document/upload
Output:
> POST /document/upload HTTP/1.1
< HTTP/1.1 201 Created
{"id":122,"documentUrl":"https://.../test-ok.pdf"}

PDF opens correctly.

External call (fails):

curl -v -X POST \
 -F "documentType=POD" \
 -F "organizationId=1005" \
 -F "accessType=test" \
 -F "file=@./test.pdf;type=application/pdf" \
 https://api.dev.testdomain.com/document/upload
Output:
* using HTTP/2
< HTTP/2 201 
{"id":124,"documentUrl":"https://.../test-bad.pdf"}

PDF is blank when opened from S3.

What we’ve tried so far

Verified the app logic — same binary works locally and in-cluster.

Added a dedicated /document location block in NGINX to handle uploads with larger body size and buffering settings.

Adjusted proxy timeouts and disabled compression.

Confirmed the app reads the file and uploads it to S3 successfully when called directly.

Despite these changes, the file still becomes blank when uploaded through the external route.

Logs

App logs (when working):

INFO: Uploading 30176 bytes to https://dev-doc.s3.us-east-1.amazonaws.com/.../test-ok.pdf

App logs (when broken):

INFO: Uploading 5758 bytes to https://dev-doic.s3.us-east-1.amazonaws.com/.../test-bad.pdf

NGINX access log (snippet):

[OK] 201 30650 189 "POST /document/upload" [BAD] 201 1048 189 "POST /document/upload"

So the request size reaching NGINX (or beyond) seems truncated when coming through API Gateway/CloudFront.

Question

Has anyone encountered a situation where multipart/form-data uploads (especially PDFs) work fine inside Kubernetes but result in truncated or blank files when routed through API Gateway / CloudFront / NGINX?

What could cause this kind of body corruption in transit?

Could it be HTTP/2 → HTTP/1.1 conversion issues?

Binary media type handling on API Gateway?

Compression or buffering in the proxy chain?

Something else?

Any insights or debugging tips would be appreciated.

• mx0: 1. Create a very small PDF file - just a magic number and some data: b"%PDF-DUMMY-DATA". 2. Upload it 3. Dump file content in fastapi logs before sending it to S3. 4. Compare it's content. This should show where the file is being corrupted. Then you can try sending bigger content until it brakes.