网页全网采集系统,是一款基于http协议的Web信息采集软件,支持集群化部署!
参考设计:http://scrapy-chs.readthedocs.org/zh_CN/latest/topics/architecture.html
单机、集群
CrawlerEngine:系统整体调度器
Scheduler:调度器/抓取队列
Downloader:下载器
Spiders:蜘蛛/新内容发现
Message Pipeline:消息管道/数据清洗
-
Notifications
You must be signed in to change notification settings - Fork 54
ljja/SmartSpiderCluster
Folders and files
| Name | Name | Last commit message | Last commit date | |
|---|---|---|---|---|
Latest commit | ||||
Repository files navigation
About
网页全网采集系统,是一款基于http协议的Web信息采集软件,支持集群化部署!
Resources
Stars
Watchers
Forks
Releases
No releases published
Packages 0
No packages published