How do I scrape a website for information?

Question

I want my program to automatically download only certain information off a website. After finding out that this is nearly impossible I figured it would be best if the program would just download the entire web page and then find the information that I needed inside of a string.

How can I find certain words/numbers after specific words? The word before the number I want to have is always the same. The number varies and that is the number I need in my program.

• BergListe: could you please post an example text?
• Vogel612: first you need to make sure, the word is unique and then you can use msdn.microsoft.com/de-de/library/…
• J. Steen: I have edited your question for clarity and used phrases more known to the community. If any of my changes were incorrect, please make edits yourself to clarify your question.
• user1017882: Your question's a bit too vague. Provide more context and some example code if you can. 'Downloading certain information off a website' is not necessarily impossible depending on the details of it. Look into screen scraping.
• Joel Peltonen: Any update on this issue?

Answer 1

Sounds like screen scraping. I recommend using CSQuery https://github.com/jamietre/CsQuery (or HtmlAgilityPack if you want). Get the source, parse as object, loop over all text nodes and do your string comparison there. The actual way of doing this varies a LOT on how the source HTML is done.

Maby something like this untested example written from memory (CSQuery)

var dom = CQ.Create(stringWithHtml);
dom["*"].Each((i, e) =>
{
 // handle only text nodes
 if (e.NodeType == NodeType.TEXT_NODE) {
 // do your check here
 }
}

Answer 2

I've used HTML Agility Pack for multiple applications and it works well. Lots of options too.

It's a lovely HTML parser that is commonly recommended for this. It will take malformed HTML and massage it into XHTML and then a traversable DOM, like the XML classes. So, is very useful for the code you find in the wild.