Multiple HTTP request in C#

Question

I need to send about 200 HTTP requests in parallel to different servers and get response. I use HttpWebRequest class in C#. But I don't see good time enhancement when requests are handled in parallel. For example if one request needs 3sec to get response, 2 request in parallel - 6sec, 3 requests - 8 secs, 4 requests - 11sec ... It is not good, I hope that there should be best time, about 10 sec for 200 requests. It looks like only 2-3 requests performs in parallel, but timeout starts immediately after WebRequest object creation. I tried set DefaultConnectionLimit and MaxServicePoints values, but id didn't help. As I understand these parameters for number of requests to one site in parallel. I need requests to different sites.

Code example that I use to test:

ArrayList a = new ArrayList(150);
for (i = 50; i < 250; i++ )
{
 a.Add("http://207.242.7." + i.ToString() + "/");
}
for (i = 0; i < a.Count; i++)
{
 Thread t = new Thread(new ParameterizedThreadStart(performRequest));
 t.Start(a[i]);
}
static void performRequest(object ip)
{
 HttpWebRequest req = (HttpWebRequest)WebRequest.Create((stirng)ip);
 HttpWebResponse resp = (HttpWebResponse)req.GetResponse();
}

Сan anyone ever encountered such a problem? Thank you for any suggestions.

• ala: I have similar issue myself, it seems that WebRequest.Create() reuse the same TCP connection for some reason. I'm wondering if you found out what the answer to your question is?

Answer 1

Instead of starting up your own threads try using the asynchronous methods of HttpWebRequest such as HttpWebRequest.BeginGetResponse and HttpWebRequest.BeginGetRequestStream.

• Nikita: I already have multithread application and need to execute this requests in already creaated threads. This is only example code. Point is that when threads peforms WebRequest.Create and GetResponse actions porfomance is very very slow. Also before HTTP request I perform TCP connection in the same threads on sever's 80 port to know port is opened or closed and it works very fast.

Answer 2

This might help - http://social.msdn.microsoft.com/forums/en-US/netfxnetcom/thread/1f863f20-09f9-49a5-8eee-17a89b591007

Suggests there is a limit on the number of TCP connections allowed, but that you can increase the limit

• ChoccyButton: More details here as well - arnosoftwaredev.blogspot.com/2006/09/… This is almost certainly the problem your hitting
• Patrick: Although.. your links seems to reference simultaneous connections to one host, and from the example in the question, the OP is making connections to several different hosts... Is this the case or does .NET have a maximum connections limit to any host as well?
• ChoccyButton: @Patrick Think you have a point. Depends I guess if all 200 of his requests go to different servers or say 10 to first server, 20 to next, 5 to next, etc. This would hit the problem. Otherwise it might not be the problem. Worth a try at least

Answer 3

Use asynchronous web requests in stead.

Edit: http://msdn.microsoft.com/en-us/library/86wf6409(VS.71).aspx

Answer 4

you can try this :

try
 {
 List<Uri> uris = new List<Uri>();
 uris.Add(new Uri("http://www.google.fr"));
 uris.Add(new Uri("http://www.bing.com"));
 Parallel.ForEach(uris, u =>
 {
 WebRequest webR = HttpWebRequest.Create(u);
 HttpWebResponse webResponse = webR.GetResponse() as HttpWebResponse;
 });
 }
 catch (AggregateException exc)
 {
 exc.InnerExceptions.ToList().ForEach(e =>
 {
 Console.WriteLine(e.Message);
 });
 }

Answer 5

This is the code of android application of sending the request. How can we use the upper code like:

params.add(new BasicNameValuePair("key", "value");
HttpPost request = new HttpPost();
List<NameValuePair> params = new ArrayList<NameValuePair>();
params.add(new BasicNameValuePair("key", "value");// How can we give the value in this format format
post.setEntity(new UrlEncodedFormEntity(params));
httpClient.execute(request);