502 Errors via Google Cloud-Run When Request Returns >64KiB JSON, Compressed?

Question

I'm having a problem with my service running under Google's Cloud-Run sending 502 responses for large API responses.

My (Angular) frontend app sends a JSON formatted request to my (Kotlin+Ktor) Cloud-Run service and gets a JSON formatted response. Everything works fine... until the response exceeds 64KiB in length, then it suddenly becomes an immediate "502 Bad Gateway" response.

There are no log messages from my service even though I have exception handlers that would show a failure. I get only this in the Cloud-Run "logs" tab:

POST 502 135 B 175 ms Chrome 126 https://my-service.com/api/LargeResponseCall

Everything looks good on my service. I've added code to log the request & response and everything appears correct meaning the problem is either in Ktor or Cloud-Run.

When I run the container on my local machine, I don't encounter any problems which tells me it's not a Ktor issues. (Unless the HTTP request created by Cloud-Run is somehow different than the one created by Chrome.)

I send plenty of other data larger than 64KiB but only the JSON response is having problems. I have configured no gateways, proxies, or intermediate servers and there is nothing special about the /api/ handlers.

Is there a limit in Cloud-Run? How can I debug this?

Update1: I changed the response type from application/json to application/octet-stream and now the large responses work just fine.

Update2: Here is a capture from Google Cloud-Trace: enter image description here

Update3: I have traced the trigger for this problem to my Ktor compression configuration:

 install(Compression) {
 gzip {
 priority = 1.0
 matchContentType(
 ContentType.Application.Json,
 )
 }
 }

When this is in place, I get 502 errors under Google Cloud-Run for uncompressed content >64KiB. I do not get them when running the container locally under Docker. If I remove this config then it works in both places, albeit without the desired compression.

Here's what I think is happening:

• Ktor, when doing compression, streams the output without setting a Content-Length header. I can see this, for responses that don't fail, on the Chrome side. No such header is sent.
• The Ktor compression (or perhaps the gzip library) is operating in 64KiB chunks. It takes 64KiB at a time from the response, compresses it, and sends it out the network.
• The Cloud-Run frontend receives the first compressed chunk but does not wait for additional data, instead failing with a 502 back to the client.

This explains why:

• Responses <64KiB work fine -- they are only one output chunk long.
• Pre-compressed gzip files are sent with a content-length, so they work fine.

One side or the other is doing something wrong here. Either Cloud-Run is not handling the missing Content-Length correctly or Ktor is not specifying something it "must" that Chrome tolerates but Cloud-Run does not.

• Roopa M: As per the update I could see your issue is resolved. Can you post your solution as answer for better community visibility.
• Brian White: It wouldn't say it's "resolved". I've "worked around the problem". I'd like to know the root cause so I can fix that (or the bug on Cloud-Run can be fixed so others don't hit this).
• Roopa M: If you are using load balancer , Can you share the statusDetails field of the load balancer logs.
• Brian White: I've never configured anything like that. The "Load Balancers", "Backends", and "Frontends" tabs are all empty. The "LB Policies" tab shows the API isn't enabled. I checked the YAML for my service and "balanc" appears nowhere. My service isn't even yet public so there isn't significant traffic that would cause anything automatic to happen.
• Roopa M: Can you share complete error logs?