This is the scrapy framework for TPDB's scraper.
Clone this repo
git clone --recurse-submodules https://github.com/ThePornDatabase/scrapy.git
Install the packages using poetry
poetry install
Next change directory to tpdb and create a copy of example settings and edit if necessary:
cd tpdb
cp settings.py.example settings.py
You can then run a scraper using scrapy crawl ScraperName
poetry run scrapy crawl Vixen
Each scraper is in it's own Python file, placed in the tpdb/spiders folder - they are stored in another repository so people can contribute.