When an object server was handling concurrent GET or POST requests
from very fast clients, it would starve other connected clients. The
greenthreads responsible for servicing the fast clients would hog the
processor and only rarely yield to another greenthread.
The reason this happens for GET requests is found in
eventlet.greenio.GreenSocket, in the send() method. When you call
.send(data) on a GreenSocket, it immediately calls .send(data) on its
underlying real socket (socket._socketobject). If the real socket
accepts all the data, then GreenSocket.send() returns without yielding
to another greenthread. Only if the real socket failed to accept all
the data (either .send(data) < len(data) or by raising EWOULDBLOCK)
does the GreenSocket yield control.
Under most workloads, this isn't a problem. The TCP connection to
client X can only consume data so quickly, and therefore the
greenthread serving client X will frequently encounter a full socket
buffer and yield control, so no clients starve. However, when there's
a lot of contention for a single object from a large number of fast
clients (e.g. on a LAN connected w/10Gb Ethernet), then one winds up
in a situation where reading from the disk is slower than writing to
the network, and so full socket buffers become rare, and therefore so
do context switches. The end result is that many clients time out
waiting for data.
The situation for PUT requests is analogous; GreenSocket.recv() seldom
encounters EWOULDBLOCK, so greenthreads seldom yield.
This patch calls eventlet.sleep() to yield control after each chunk,
preventing any one greenthread's IO from blocking the hub for very
long.
This code has the flaw that it will greenthread-switch twice when a
send() or recv() does block, but since there isn't a way to find out
if a switch occurred or not, there's no way to avoid it. Since
greenlet switches are quite fast (faster than system calls, which the
object server does a lot of), this shouldn't have a significant
performance impact.
Change-Id: I8549adfb4a198739b80979236c27b76df607eebf