1
1
Fork
You've already forked crawler-server
0
Server for the Mwmbl crawler
This repository has been archived on 2024年08月10日. You can view files and clone it, but you cannot make any changes to its state, such as pushing and creating new issues, pull requests or comments.
  • Python 91%
  • Dockerfile 9%
Find a file
Daoud Clarke e1bdeb95a8 Typo
2022年06月23日 00:54:25 +01:00
crawler_server Typo 2022年06月23日 00:54:25 +01:00
.dockerignore Fix docker build 2022年01月18日 22:03:13 +00:00
.env Update .env 2022年03月29日 16:53:06 +02:00
.gitignore Update .gitignore 2022年03月29日 16:52:20 +02:00
docker-compose.yml Added docker-compose 2022年03月29日 16:47:04 +02:00
Dockerfile Specify correct host 2022年01月18日 22:47:49 +00:00
poetry.lock WIP: store crawled URLs in a database to prevent duplicate effort 2022年05月25日 22:21:08 +01:00
pyproject.toml WIP: store crawled URLs in a database to prevent duplicate effort 2022年05月25日 22:21:08 +01:00
README.md Update README.md 2022年03月29日 16:57:10 +02:00

Mwmbl Crawler Server

The Mwmbl crawler is a distributed crawler with a central server. The central server collects batches of crawled data from the clients and stores it in long term storage for indexing and analysis.

Docker setup

Clone this repository git clone https://github.com/mwmbl/crawler-server then enter cd crawler-server To start crawler server via docker first edit .env file and then run docker-compose up -d To rebuild image execute docker-compose build