html scrape in .net

Question

In .net what is the best way to scrape HTML web pages.

Is there something open source that runs on .net framework 2 and and put all the html into objects. I have read about "HTML Agility Pack" but is there any think else?

• ThiefMaster: Why did you tag this with c# and vb.net?
• nyxthulhu: Are you trying to scape pages, or process pages? Do you need to look at the contextual information from the DOM or just Spider to duplicate?
• Hello-World: I want the vb.net to open the page and look at the html the take what it needs. I thought .net as it has more power than javscript and the sites wont be on my server
• dtsg: What are you looking to extract from the HTML you get back?
• Govind Malviya: developer.mindtouch.com/SgmlReader

Answer 1

I think HtmlAgilityPack is but you can also use

1. Fizzler : css selector engine for C#
2. SgmlReader : Convert html to valid xml
3. SharpQuery : Alternative of fizzler

Answer 2

You might use Tidy.net, which is a c# wrapper for the Tidy Library to convert HTML in XHTML available here: http://sourceforge.net/projects/tidynet/ so you could get valid XML and process it as such.

I'd make it this way:

 // don't forget to import TidyNet and System.Xml.Linq
 var t = new Tidy();
 TidyMessageCollection messages = new TidyMessageCollection();
 t.Options.Xhtml = true;
 //extra options if you plan to edit the result by hand
 t.Options.IndentContent = true;
 t.Options.SmartIndent = true;
 t.Options.DropEmptyParas = true;
 t.Options.DropFontTags = true;
 t.Options.BreakBeforeBR = true;
 string sInput = "your html code goes here";
 var bytes = System.Text.Encoding.UTF8.GetBytes(sInput);
 StringBuilder sbOutput = new StringBuilder();
 var msIn = new MemoryStream(bytes);
 var msOut = new MemoryStream();
 t.Parse(msIn, msOut, messages);
 var bytesOut = msOut.ToArray();
 string sOut = System.Text.Encoding.UTF8.GetString(bytesOut);
 XDocument doc = XDocument.Parse(sOut);
 //process XML as you like

Otherwise, HTML Agility pack is ok.