Windows CPU Scheduling, "Post-Wireshark" UDP receive latency?

Question

I have developed a C# program that receives UDP packets and writes the received data to a queue on thread A. The program reads from the queue, performs some data processing, and then sends another UDP packet with the modified data on thread B. Here is a simplified portion of my code:

public void Task ListeningThread()
{
 while (true)
 {
 int bytesReceived = mySocket.ReceiveFrom(buffer, ref RemoteEP);
 logfile.WriteLine("dtReceived: " + (DateTimeOffset.Now - previousreceiveTime).ToString(@"ss\.ffff"));
 previousreceiveTime = DateTimeOffset.Now;
 if (bytesReceived > 0)
 {
 byte[] receivedPacket = new byte[bytesReceived];
 Array.Copy(buffer, receivedPacket, bytesReceived);
 packetQueue.Add(receivedPacket);
 }
 }
}
public void Task ProcessingThread()
{
 byte[] array = Enumerable.Repeat((byte)0x20, 500).ToArray(); //For testing
 while (true)
 {
 foreach (var packet in packetQueue.GetConsumingEnumerable())
 {
 Socket.SendTo(array, SocketFlags.None, new IPEndPoint(IPAddress.Parse("172.16.108.162"), 565));
 dttw.WriteLine("dtSent: " + (DateTimeOffset.Now - previousreceiveTime).ToString(@"ss\.ffff") + " Data: " + array.data);
 }
 }
}

I am receiving UDP packets at 100 Hz, so when things are working correctly the logfile and associated wireshark capture look something like this:

dtReceived: 00.0100
dtSent: 00.0007 Data: 124
dtReceived: 00.0100
dtSent: 00.0009 Data: 125
dtReceived: 00.0099
dtSent: 00.0007 Data: 126
dtReceived: 00.0099
dtSent: 00.0009 Data: 127
dtReceived: 00.0100
dtSent: 00.0011 Data: 128

enter image description here

(.161 is my local machine, .163 is the machine that I am receiving data from, and .162 is the machine that I am sending data to.)

However, occasionally the following behavior occurs:

dtReceived: 00.0100
dtSent: 00.0006 Data: 82
dtReceived: 00.0099
dtSent: 00.0006 Data: 83
dtReceived: 00.0607 //<-- Problem here, data should have been received after 0.0100 seconds 
dtReceived: 00.0000 //<-- 5 more messages have stacked up in the mean-time
dtReceived: 00.0000
dtReceived: 00.0000
dtReceived: 00.0000
dtReceived: 00.0000
dtSent: 00.0007 Data: 84 //<-- After the 6 packets arrive near-simultaneously, 6 packets are sent in quick succession
dtSent: 00.0026 Data: 85
dtSent: 00.0037 Data: 86
dtSent: 00.0047 Data: 87
dtSent: 00.0056 Data: 88
dtSent: 00.0064 Data: 89
dtReceived: 00.0091
dtSent: 00.0007 Data: 90
dtReceived: 00.0100
dtSent: 00.0006 Data: 91

enter image description here

The Wireshark capture indicates that the incoming are arrive at the correct rate. However, the packets are stacking up before finally arriving at the application, at which time all of the incoming packets are received and processed nearly simultaneously. This is running on a Windows 11 system. Neither the CPU nor the network adapter is anywhere close to being taxed. Does anyone have an idea of what could be causing this or a potential solution?

• Tangentially Perpendicular: Windows is not a real time operating system and it doesn't guarantee a response time. Almost anything could be happening: some system activity; some scheduled task; or just a lengthy despatch of some other process. If you want a solution you need to explain why it's a problem.
• Jake: The system that is receiving the processed data (.162) requires an update corresponding to every message from the original sender (.163). These updates must happen before .163 sends its next message. An occasional packet drop would not be a huge deal but the above issue happens quite a bit.
• Tangentially Perpendicular: How does 162 know that 163 is sending an update? The updates are clearly queued at 161, so they'll be received in order at 162. If there's a time-sensitive element to this you need to explain more about the system. Focusing on the response time of a single Windows system is unlikely to yield a result, unless you can consider replacing it with an RTOS
• Charlieface: else { Thread.Sleep(1); why is this here, why are you going to sleep if you need to wait for more packets? Why are you (clearly) using a ConcurrentQueue and not either a BlockingCollection.GetConsumingEnumerable or a PipeReader/PipeWriter pair like learn.microsoft.com/en-us/dotnet/standard/io/pipelines which would be a much better fit. You should also be using async to better handle async IO flows.
• Charlieface: No there isn't any such buffering, it should be near instantaneous, that is to say that as soon as a UDP datagram comes in then the waiting threadpool thread will be scheduled on the kernel's running queue. As long as the CPU isn't busy it should be executed almost immediately. Please show your current version of the code, wthout Thread.Sleep or any other such delaying mechanism. What does api.SendModifiedData do, please show.

Answer 1

I believe the problem here is related to the Windows CPU scheduling on the laptop that I am running this code on, as @Tangentially Perpendicular suggested. This is a "corporately managed" laptop with good hardware (13th gen i9 processor, 64 Gb RAM) but a lot of control software running in the background. This exact code ran on an unmanaged, offline laptop without any of the issues described above.