Sending device data of 10k items with Parallel.ForEach

Question

I have a deviceList of more than 10k items and want to send data by calling another method.

I tried to use Parallel.Foreach but I'm not sure is this the correct way to do it.

I have published this webapp on azure, I have tested this for 100 it works fine but for 10k it got timeout issue.

This is working code , I need a improvement here :)

private List<Task> taskEventList = new List<Task>();
public async Task ProcessStart()
{
 string messageData = "{\"name\":\"DemoData\",\"no\":\"111\"}";
 RegistryManager registryManager;
 Parallel.ForEach(deviceList, async (device) =>
 {
 // get details for each device and use key to send message
 device = await registryManager.GetDeviceAsync(device.DeviceId);
 SendMessages(device.DeviceId, device.Key, messageData);
 });
 if (taskEventList.Count > 0)
 {
 await Task.WhenAll(taskEventList);
 }
}
private void SendMessages(string deviceId, string Key, string messageData)
{
 DeviceClient deviceClient = DeviceClient.Create(hostName, new DeviceAuthenticationWithRegistrySymmetricKey(deviceId, deviceKey), Microsoft.Azure.Devices.Client.TransportType.Mqtt);
 //created separate Task
 var taskEvents = Task.Run(() => ProcessMessages(deviceId, string messageData));
 taskEventList.Add(taskEvents);
}
private async Task ProcessMessages(string deviceId, string messageData)
{
 var startTime = DateTime.UtcNow;
 while (DateTime.UtcNow - startTime < TimeSpan.FromMinutes(15))
 {
 await deviceClient.SendEventAsync(messageData);
 }
}

• Adrian Iftode: You are probably getting into a thread starvation as you just scheduling tasks and also borrowing threads from the thread pool. You need some kind of batching or throttling, as the thread pool is limited ( a thread has 2 MB, imagine 10k requests for threads)
• Adrian Iftode: Also if DeviceClient.Create creates and disposes a new HttpClient everytime, then you'll get out of available sockets
• Vogel612: Hey @AdrianIftode that seems to be the beginnings of an answer. Are you sure you don't want to write one, now that the post has been reopened?
• Adrian Iftode: Thanks, I would like also to know about DeviceClient.Create, possible @Neo ?
• Adrian Iftode: @Neo, are you sure everything is ok with this check: while (DateTime.UtcNow - startTime < TimeSpan.FromMinutes(15)) ?