Run again with 480 engines, 20 tasks/engine (top is main, 3093386f , bottom is this pr, a98a4fa). Workload is load-balanced submission of random 0-1s tasks (same seed), 20 tasks/engine for a total of 9600 tasks.

Screen Shot 2021年07月21日 at 14 25 20

Can see that while the client is working to produce the tasks, there is still contention between serializing in the main thread and actually sending in the io thread until the main thread is done (purple line). This completes 1s faster in this PR (7.6s vs 8.4s). The first result doesn't arrive for 2 more seconds, which is really around when the last real send completes and receives start being processed.

The bubble can be seen around 11s in main, which is where sends and receives are both being processed, and this is gone after this PR.

Originally posted by @minrk in #534 (comment)