How to scrape data from web page using asp.net

Question 1

I want to get values from my html page. I tried to get the same using HttpWebRequest but i am not able to do so far please help ?

 <div class="container">
 <div class="one-third column">
<ol start="181">
<li><a href="/lyrics/hindi-lyrics-of-Aaye%20Din%20Bahar%20Ke.html">Aaye Din Bahar Ke</a>
</li><li><a href="/lyrics/hindi-lyrics-of-Aayega%20Aane%20Wala.html">Aayega Aane Wala</a>
</li><li><a href="/lyrics/hindi-lyrics-of-Aayi%20Milan%20Ki%20Raat.html">Aayi Milan Ki Raat</a>
</li><li><a href="/lyrics/hindi-lyrics-of-Aiyyaa.html">Aiyyaa</a>
</li><li><a href="/lyrics/hindi-lyrics-of-Ajab%20Gazabb%20Love.html">Ajab Gazabb Love</a>
</li></ol>
 </div>
<div class="sixteen columns">
<hr>
More Pages: 
<a href="hindi-songs-starting-A.html">1</a> : <a href="hindi-songs-starting-A-page-2.html">2</a> : 3 : <a href="hindi-songs-starting-A-page-4.html">4</a> : <a href="hindi-songs-starting-A-page-5.html">5</a> : <a href="hindi-songs-starting-A-page-6.html">6</a> : 
 <hr>
<center>
<h4>Hindi Lyrics By Movie Title</h4>
<p> 
<a href="/lyrics/hindi-songs-starting-0.html">0-9</a>
<a href="/lyrics/hindi-songs-starting-A.html">A</a>
<a href="/lyrics/hindi-songs-starting-B.html">B</a>
<a href="/lyrics/hindi-songs-starting-W.html">W</a>
X
<a href="/lyrics/hindi-songs-starting-Y.html">Y</a>
<a href="/lyrics/hindi-songs-starting-Z.html">Z</a>
 | <a href="http://www.hindilyrics.net/songs/">Top Songs</a>
</p>
</center>
 </div>

this is my html i want to get all links

Question 2

what you have tried. share your code(C#)

Question 3

we can scrape we sit Using htmlagilitypack. you can download from here http://htmlagilitypack.codeplex.com/

string urls = "your web page";
 string result = string.Empty;
 HttpWebRequest request = (HttpWebRequest)WebRequest.Create(urls);
 request.UserAgent = @"Mozilla/5.0 (Windows; U; Windows NT 6.1; en-US; rv:1.9.1.5) Gecko/20091102 Firefox/3.5.5";
 using (var stream = request.GetResponse().GetResponseStream())
 using (var reader = new StreamReader(stream, Encoding.UTF8))
 {
 result = reader.ReadToEnd();
 }
 HtmlAgilityPack.HtmlDocument doc = new HtmlAgilityPack.HtmlDocument();
 doc.Load(new StringReader(result));
 var elements = doc.DocumentNode.SelectNodes("//div[@class='one-third column']");
 foreach (HtmlNode item in elements)
 {
 var node1 = item.SelectNodes(".//li");
 foreach (HtmlNode li in node1)
 {
 var a = li.SelectSingleNode("//a").Attributes["href"].Value;//your link
 }
 }

Question 4

You can use a System.Net.Http.HttpClient class and it's GetAsync() method. HttpClient class comes with nice features for asynchronous downloading of websites. Or you can use WebRequest class - a very basic approach.

Prithvi Raj Nandiwal 3,3043 gold badges24 silver badges34 bronze badges · Accepted Answer · 2014-12-29 12:16:33Z

we can scrape we sit Using htmlagilitypack. you can download from here http://htmlagilitypack.codeplex.com/

string urls = "your web page";
 string result = string.Empty;
 HttpWebRequest request = (HttpWebRequest)WebRequest.Create(urls);
 request.UserAgent = @"Mozilla/5.0 (Windows; U; Windows NT 6.1; en-US; rv:1.9.1.5) Gecko/20091102 Firefox/3.5.5";
 using (var stream = request.GetResponse().GetResponseStream())
 using (var reader = new StreamReader(stream, Encoding.UTF8))
 {
 result = reader.ReadToEnd();
 }
 HtmlAgilityPack.HtmlDocument doc = new HtmlAgilityPack.HtmlDocument();
 doc.Load(new StringReader(result));
 var elements = doc.DocumentNode.SelectNodes("//div[@class='one-third column']");
 foreach (HtmlNode item in elements)
 {
 var node1 = item.SelectNodes(".//li");
 foreach (HtmlNode li in node1)
 {
 var a = li.SelectSingleNode("//a").Attributes["href"].Value;//your link
 }
 }

CollectivesTM on Stack Overflow

How to scrape data from web page using asp.net

2 Answers 2

Comments

Comments

Your Answer

Sign up or log in

Post as a guest

Post as a guest

Hot Network Questions

CollectivesTM on Stack Overflow

2 Answers 2

Comments

Comments

Your Answer

Sign up or log in

Post as a guest

Post as a guest

Related