3 Open
1 Closed
It's often the case that a torrent needs to be replaced or updated for one reason or another! Let's make a system for automating that, detecting repacked torrents, as well as laying the groundwork for a system for relating torrents to one another to e.g. indicate that one is an incremental update of some base dataset, is the next version, and so on. Descriptions in the issues.
7 Open
24 Closed
the task is essentially this: we have this dataset, and it is structured as batches. we need to split that scraping task up because the server has some rate limits and it might be taken down before we can scrape it all.
This is also a general need for us, to be able to
- declare a dataset,
- split it into parts,
- write a crawler for the parts (ideally eventually we would be able to do this via the site for simple crawls that are just recursing under some URL),
- mark them as in progress,
- scrape a part,
- repeat until all parts scraped,
- and then pack them into torrents and upload,
- matching them to the relevant dataset part
0 Open
0 Closed