Performing concurrently http requests using httpclient in .NET C#

Question

I have made a console application which basically performs large amount of requests towards my server. 10-15000 requests per user.

So I've written this code which uses .NET's HTTP Client library:

public async Task<string> DoRequest(string token)
{
var request = new HttpRequestMessage(HttpMethod.Post, "mysite.com");
string requestXML = "my xml request goes her...";
request.Content = new StringContent(requestXML, Encoding.UTF8, "text/xml");
var response = await httpClient.SendAsync(request);
return await response.Content.ReadAsStringAsync();
}

So now I'm trying to perform as much as HTTP requests that I can per 1 second... I don't know whats the upper limit of this so I've used parallel for loop to speed this:

Parallel.For(0,list.Count(),items=>{
DoRequest(item.Token);
});

But I'm not very happy with the speed of the code and how the requests are being made...

Is there any way that I can speed things up n do maybe up to 10-100 requests per 1 second ? Or what is the maximum limit ?

Can someone help me out ?

P.S. guys I created only one instance of httpclient class because I read that it's a good practice to make only one since each time a new object of this class is made the connection gets closed, which is not what we want no ?

• DavidG: Your test might not be particularly useful. For example, HttpClient is restricted to the number of connections to the same server at any one time (though you can change that with ServicePointManager.DefaultConnectionLimit)
• User987: @DavidG can you show me exactly how to do this and where? In this same method or ?
• Botonomous: ServicePointManager.DefaultConnectionLimit is a property @User987 just set it.

Answer 1

Is there any way that I can speed things up n do maybe up to 10-100 requests per 1 second?

That's an unanswerable question.

Or what is the maximum limit?

As David noted, you need to set ServicePointManager.DefaultConnectionLimit. This should suffice as the first line in your Main method:

ServicePointManager.DefaultConnectionLimit = int.MaxValue;

I've used parallel for loop to speed this

That's the incorrect tool for this job. You do want concurrency, but not parallelism. Asynchronous concurrency is expressed using Task.WhenAll:

var tasks = list.Select(item => DoRequest(item.Token));
await Task.WhenAll(tasks);

I created only one instance of httpclient class because I read that it's a good practice to make only one since each time a new object of this class is made the connection gets closed, which is not what we want no?

That is correct.

• User987: Stephen, and how would I actually load the results of Task.WhenAll(tasks) ? Inside each of task theres an XML response. How would I load them into and XMLDocument then ?
• Botonomous: @User987 Chain continuation tasks to them. Have the continuation read the XML and do whatever you want with it after that.
• Stephen Cleary: @User987: var results = await Task.WhenAll(tasks); will get you a results variable which is an array of XML responses.
• user2966445: @Stephen Cleary How would you do this but limit the number of concurrent connections?
• Stephen Cleary: You can use DefaultConnectionLimit or SemaphoreSlim (there are several examples on SO).