Crawler

A web crawler is a program, often called a bot or robot, which systematically browses the Web to collect data from webpages. Typically search engines (e.g., Google, Bing, etc.) use crawlers to build indexes.

See also

• Related glossary terms:
  • Search engine
• Web crawler on Wikipedia