The Web Data API for AI - Turn entire websites into LLM-ready markdown or structured data 🔥
markdown crawler data scraper ai html-to-markdown web-crawler scraping webscraping rag llm ai-scraping
-
Updated
Sep 1, 2025 - TypeScript
#
web-crawler
Here are 1,070 public repositories matching this topic...
Language: All
Filter by language
Sort: Most stars
Sort options
Python scraper based on AI
markdown crawler ai html-to-markdown web-crawler scraping web-scraping rag automated-scraper scraping-python web-crawlers llm ai-scraping
-
Updated
Aug 13, 2025 - Python
Crawlee—A web scraping and browser automation library for Node.js to build reliable crawlers. In JavaScript and TypeScript. Extract data for AI, LLMs, RAG, or GPTs. Download HTML, PDF, JPG, PNG, and other files from websites. Works with Puppeteer, Playwright, Cheerio, JSDOM, and raw HTTP. Both headful and headless mode. With proxy rotation.
nodejs javascript npm crawler scraper automation typescript web-crawler headless scraping crawling web-scraping web-crawling headless-chrome apify puppeteer playwright
-
Updated
Sep 1, 2025 - TypeScript
Distributed web crawler admin platform for spiders management regardless of languages and frameworks. 分布式爬虫管理平台,支持任何语言和框架
go docker platform crawler spider web-crawler scrapy webcrawler scrapyd-ui webspider crawling-tasks crawlab spiders-management
-
Updated
Aug 16, 2025 - Go
新一代爬虫平台,以图形化方式定义爬虫流程,不写代码即可完成爬虫。
-
Updated
Jun 14, 2023 - Java
A collection of awesome web crawler,spider in different languages
-
Updated
Jun 16, 2024
Ingest, parse, and optimize any data format ➡️ from documents to multimedia ➡️ for enhanced compatibility with GenAI frameworks
-
Updated
Jun 11, 2025 - Python
Crawlee—A web scraping and browser automation library for Python to build reliable crawlers. Extract data for AI, LLMs, RAG, or GPTs. Download HTML, PDF, JPG, PNG, and other files from websites. Works with BeautifulSoup, Playwright, and raw HTTP. Both headful and headless mode. With proxy rotation.
python crawler scraper automation web-crawler headless scraping crawling pip web-scraping beautifulsoup web-crawling hacktoberfest headless-chrome apify playwright
-
Updated
Sep 1, 2025 - Python
🔥 Official Firecrawl MCP Server - Adds powerful web scraping to Cursor, Claude and any other LLM clients.
web-crawler web-scraping data-collection batch-processing content-extraction search-api claude llm-tools firecrawl model-context-protocol mcp-server firecrawl-ai javascript-rendering
-
Updated
Aug 27, 2025 - JavaScript
Cross Platform C# web crawler framework built for speed and flexibility. Please star this project! +1.
c-sharp unit-testing crawler spider csharp parsing cross-platform web-crawler netcore pluggable spiders csharp-library abot netcore2 netstandard20 netcore3 javascript-renderer netstandard21 abot-nuget netsta
-
Updated
Sep 9, 2024 - C#
The Ultimate Information Gathering Toolkit
osint web-crawler whois-lookup virustotal information-gathering server-info dns-lookup reconnaissance cms-detection recon-tools email-harvester ssl-analitcs directory-finder txt-records pastebin-monitoring
-
Updated
Oct 8, 2024 - Python
简单易用的Python爬虫框架,QQ交流群:597510560
-
Updated
Jun 10, 2022 - Python
Internet search engine for text-oriented websites. Indexing the small, old and weird web.
java search-engine web-crawler indexer language-processing web-scale internet-search no-cloud self-hostable small-web alt-search
-
Updated
Sep 1, 2025 - HTML
CLI tool for saving a faithful copy of a complete web page in a single HTML file (based on SingleFile)
nodejs cli dockerfile crawler web-crawler archiving web-scraper web-scraping web-archiving scraping-websites single-file deno
-
Updated
Jun 2, 2025 - JavaScript
A scalable, mature and versatile web crawler based on Apache Storm
-
Updated
Sep 1, 2025 - Java
PulsarRPA: An AI-Enabled, Super-Fast, Thread-Safe Browser Automation Solution! 💖
web-crawler web-scraper web-scraping dom-manipulation web-crawling browser-automation dom-api ai-agents web-extraction rpa web-extractor llm ai-crawler text-to-action browser-use ai-rpa ai-browser-control
-
Updated
Jul 22, 2025 - Kotlin
Run a high-fidelity browser-based web archiving crawler in a single Docker container
-
Updated
Aug 30, 2025 - TypeScript
A versatile Ruby web spidering library that can spider a site, multiple domains, certain links or infinitely. Spidr is designed to be fast and easy to use.
-
Updated
Jun 30, 2025 - Ruby
Improve this page
Add a description, image, and links to the web-crawler topic page so that developers can more easily learn about it.
Add this topic to your repo
To associate your repository with the web-crawler topic, visit your repo's landing page and select "manage topics."