Or it will also be triggered if the remote peer will close the socket as well.

ng too much time after the final close. It might be that something like an io_uring request is holding a reference to the socket that would prevent the close from terminating the connection.

Let me test shutdown(2) and come back to you... in the meantime I also checked for your other theory (holding a reference to the socket) and at least this seems to be not true. I verified that I received a notification for all previous submitted ops so there shouldn't be anything pending at all.

Ok also tried to use shutdown and its the same.. Without using a low value for TCP_USER_TIMEOUT it basically waits forever for the second notification even after the result of shutdown (or close) is observed. Like said before all other completions are received as well so the only one that is not received yet is the second notification for the send_zc that signals that the buffers can be reused. 🤷 As close should trigger a RST I would have expected that the buffers could be re-used directly as after the RST is send there should be no need to wait for any TCP ack at all.

Also as a side note this problem only shows up when the closure is trigged locally. If the remote peer closes the connection by calling close I will receive the second notification without any waiting time and so can reuse the buffer.

Yeah, it was brain fog on my part. It makes perfect sense from your investigation, the socket just tries to send out everything it has queued. Does TCP_USER_TIMEOUT work for your case? There is not much kernel can do apart from forcing it. And if you have a better way to detect such uncooperative sockets, I guess you can set TCP_USER_TIMEOUT=1 at the right moment.

Yeah TCP_USER_TIMEOUT works... that said I am still a bit confused why after a close (which triggers a TCP RST) we even need to try to send out stuff that is queued. Shouldn't we just drop stuff on the floor at this point and so give the memory back to the system as fast as possible ?

@isilence any idea ?

It's not like it's something that can be changed without breaking users. I can see why it might've been done the current way, e.g. users would need a way to wait for the last send to be sent and/or acked before close or so. It can be handled somehow, but thinking about it, closing reliably is surprisingly not such an easy problem.