Can you add an async version of the benchmark? Having both APIs be tested before merging this into the backpressure branch would be ideal.

Done. Added the scrip to the Jira ticket. The async version still shows a significant reduction in the number of wasted retries and lower p50 to p90 latency but less/no benefit for p99 and p100 latency.

Before:

$ python3.13 repro-storm-async.py
Completed 200 concurrent async transactions in 3.467694044113159 seconds
Total retry attempts: 5950
avg latency: 1.87s p50: 2.04s p90: 3.22s p99: 3.45s p100: 3.46s

After:

$ python3.13 repro-storm-async.py
Completed 200 concurrent async transactions in 3.5634748935699463 seconds
Total retry attempts: 887
avg latency: 1.48s p50: 1.41s p90: 2.81s p99: 3.49s p100: 3.56s

Done. Added the scrip to the Jira ticket. The async version still shows a significant reduction in the number of wasted retries and lower p50 to p90 latency but less/no benefit for p99 and p100 latency.

Before:

$ python3.13 repro-storm-async.py
Completed 200 concurrent async transactions in 3.467694044113159 seconds
Total retry attempts: 5950
avg latency: 1.87s p50: 2.04s p90: 3.22s p99: 3.45s p100: 3.46s

After:

$ python3.13 repro-storm-async.py
Completed 200 concurrent async transactions in 3.5634748935699463 seconds
Total retry attempts: 887
avg latency: 1.48s p50: 1.41s p90: 2.81s p99: 3.49s p100: 3.56s

Async sees significantly less improvement with the backoff, but I'd say that's expected. Asyncio's cooperative multitasking structure already prevents a given operation from retrying before the other concurrent async tasks have had a chance to run (assuming the async/await code is written correctly).