What is causing a "serverRuntime@appConcurrentRequestLimit setting is being exceeded" error?

Question

Can anyone provide some suggestions for where we can look to pinpoint the cause for this issue?

• We are experiencing an issue where we are getting this error from IIS: "serverRuntime@appConcurrentRequestLimit setting is being exceeded"

• Our site stops accepting requests and end-users are shown a "503 Service Unavailable" error

• We have a load balancer that manages the connections between 3 servers

• We have been able to anticipate that the issue is about to happen, by watching if the count reaches 5000 for the "Pipeline Instance Count" in the performance monitor for each server

• High traffic does not seem to be the cause, because we've experienced the issue both during high traffic with 4500 calls/min and during lower traffic with about 1500 calls/min

• Recycling our application pool on all 3 servers temporarily fixes the issue

• Our application pool is also automatically recycled every 4am

• YurongDai: You can use Windows Performance Monitor (PerfMon) to monitor your server's CPU and memory usage. High memory usage or memory leaks can reduce the number of concurrent requests that the server can handle.
• Nicole Uy: Thanks @YurongDai! I'll see if the memory usage looks out of the ordinary