multithreading program with webrequests c#

Question

I want to make multithreading asynchronous download manager. But i have problems with multithreading. One thread correctly works, but when I create second thread - works nothing. I assume that a problem with synchronism of webrequest. I read this answer Multithreading a large number of web requests in c#, but I didn't understand completely. Now question: How can I modify a code to use a multithreading(Thread, Threadpool).

class DownloadableContent:

{
 private string url { get; set; }
 private string path { get; set; }
 private Stream streamResponse { get; set; }
 private Stream streamLocal { get; set; }
 private HttpWebRequest webRequest { get; set; }
 private HttpWebResponse webResponse { get; set; }
 public DownloadableContent(string url, string path)
 {
 this.url = url;
 this.path = path;
 }
 public void Download()
 {
 using (WebClient wcDownload = new WebClient())
 {
 try
 {
 webRequest = (HttpWebRequest)WebRequest.Create(url); 
 webRequest.Credentials = CredentialCache.DefaultCredentials;
 webResponse = (HttpWebResponse)webRequest.GetResponse();
 Int64 fileSize = webResponse.ContentLength;
 streamResponse = wcDownload.OpenRead(url);
 streamLocal = new FileStream(path, FileMode.Create, FileAccess.Write, FileShare.None);
 byte[] downBuffer = new byte[2048]; 
 int bytesSize = 0;
 while ((bytesSize = streamResponse.Read(downBuffer, 0, downBuffer.Length)) > 0)
 {
 streamLocal.Write(downBuffer, 0, bytesSize);
 }
 }
 finally
 {
 streamResponse.Close();
 streamLocal.Close();
 }
 }
 } 
}

And class main:

DownloadableContent file = new DownloadableContent("url", @"path"); 
Thread thread = new Thread(file.Download);
thread.Start();

• Jim Mischel: Your code is very confusing. You're using WebClient and HttpWebRequest. As a result, you're making two requests for every URL. You should remove the webRequest and webResponse, and just use WebClient.
• Jim Mischel: Very likely the reason you're having trouble is that you're not disposing of the webRequest and webResponse objects that you're creating. I know from experience that this often causes things to stop working after just a few requests. Since you're not using them, just remove them from your code.
• Kuchur Andrei: I left in a code only Webclient , and now it works good

Answer 1

The best advice that I can give for you, is to use TPL. It's one good library from Microsoft to manager threads. I had used this for a similar problems of yours in my code, basically I had to download 8000 urls, in the begin the normal process was taking 30 minutes. After I used this library, the same process was finished in 30 seconds.

TPL LINK

Task Parallel Library (TPL)

Please, get a look in the example:

BookStore- GitHub

Answer 2

It is worth for you to read Async programming then take a look at this http://msdn.microsoft.com/en-us/library/System.Net.WebClient_methods(v=vs.110).aspx You have method to download thing asynchronously. Also take a look at TPL and avoid threads http://msdn.microsoft.com/en-us/library/dd460717(v=vs.110).aspx

Probably reading a little more will help you avoid lots of headaches.

This is an easy example

private static void Main(string[] args)
{
 var urlsAndPaths = new Dictionary<string, string>();
 urlsAndPaths.Add("http://i.forbesimg.com/media/lists/people/lionel-messi_416x416.jpg","messi.jpg");
 urlsAndPaths.Add("http://sizzlingsuperstars.com/wp-content/uploads/2014/07/Cristiano-Ronaldo-2-480x309.jpg", "cristiano.jpg"); 
 foreach (var kvp in urlsAndPaths)
 {
 var wc = new WebClient();
 wc.DownloadFileAsync(new Uri(kvp.Key),kvp.Value);
 }
 Console.ReadKey();
}

• Kuchur Andrei: thanks for answer, looks correctly, but program condition is "New Download in New Thread" (training task) using Thread and Threadpool. (two differents realizations)
• Kuchur Andrei: This is tasks for novices in .NET :)

Answer 3

Depending on which version of the .NET framework you are using, you could take advantage of the Task class. You could do something like this

 foreach (var uri in myCollection)
 {
 Task.Factory.StartNew(() =>
 {
 try
 {
 using (System.Net.WebClient client = new System.Net.WebClient())
 {
 client.DownloadFileCompleted += (o, args) =>
 {
 //Do something with the download
 };
 client.DownloadFileAsync(uri);
 }
 }
 catch (Exception ex)
 {
 //Do something
 }
 });
 }

Answer 4

Are you on .NET 4.5? The "new way" to do this is with Task.WhenAll, which gets your downloads going asynchronously and in parallel but allows the framework to decide if/when work should be scheduled to the thread pool.

var client = new System.Net.WebClient();
await Task.WhenAll(urls.Select(url => {
 var path = ?? // build local path based on url?
 return client.DownloadFileAsync(url, path);
});