| LICENSE | Initial commit | |
| README.md | Update README.md | |
torrent-ranger
Create a torrent of a public data directory listing, without needing to fully download and seed.
Do you want to help in the effort of finding, downloading and sharing our public scientific data, but don't have terabytes of storage or a gigabit internet connection? Then the torrent-ranger is for you!
Designed for people with high passion and drive for data rescue, but low resources, torrent-ranger allows you to act as a sort of "scout" for large-scale, simple directory listings. With a single CLI command, you can create a torrent-mirror of the files in a directory, which you can then post to your favorite torrent site. Then, users can download the first copy of the data into the new swarm (first seed) using the torrent's included webseed, which uses the server itself to upload the data, rather than your own bandwidth. E voila! The data has been preserved in a decentralized, robust, P2P framework, reliant on no single server or government/university portal, regardless of size.
How it works:
- You build your CLI command, including the url of the directory (or subdirectory) you wish to save. Select your concurrency, trackers, piece size, torrent type (v1 or hybrid), etc., and run it.
- The ranger recursively crawls the directory and creates a local manifest of all file paths in the directory, their sizes and other info.
- It then streams piece-sized chunks of the data via range requests, downloading concurrent chunks and hashing them before dumping them (as the piece hashes are all we need), and adds them to a hashes.bin file.
- The torrent is then built (using libtorrent under the hood) using the url itself as the webseed, trackers (selectable but with curated, recommended defaults), the manifest and the piece hashes in hashes.bin.
After the torrent is first posted, and users begin to download it, the data is served in a classic HTTP streaming direct download. But, once that first copy is fully in the swarm, the Bittorrent protocol prioritizes P2P piece sharing, and in that moment the torrent is like any other, and the server is simply a fallback peer that shares pieces only if no one else is serving them up.
CLI
Install
with pipx (recommended)
pipx install torrent-ranger
or with pip (user site)
pip install --upgrade torrent-ranger
Run it as the console script 'torrent-ranger' (if installed with an entry point) or via Python:
torrent-ranger https://example.org/data/ # preferred python -m torrent_ranger https://example.org/data/ # fallback
The URL must end with a trailing slash and should be an HTTP/HTTPS directory listing.
Usage
text torrent-ranger URL [options]
Resulting files live under:
// ├─ manifest.jsonl ├─ hashes.bin └─ .torrent
(Outputs are derived from the URL slug and last path segment.)
Options
| Option | Type / Default | Description |
|---|---|---|
| url (positional) | string | Base HTTP directory URL ending with /. |
| -o, --output-root | Path, default: ./datasets (under CWD) | Root folder to store the dataset, manifest, hashes, and torrent. |
| -p, --piece-length | MiB (int), default: auto | Piece length. If provided, interpreted as MiB (see note). |
| -c, --concurrency | int, default: 16 | Max parallel HTTP requests during hashing. |
| -t, --trackers | list of URLs, default: curated | Tracker announce URLs (space-separated). |
| --comment | string, default: Snapshot taken Z | Optional torrent comment; default is set at build time. |
| --v1-only | flag | Produce v1-only torrent (no v2/hybrid). |
| --skip-snapshot | flag | Skip taking a snapshot and use existing manifest.jsonl. |
| --skip-hash | flag | Skip hashing and use existing hashes.bin. |
| --resume | flag | Convenience flag: sets --skip-snapshot and resumes hashing from existing hashes.bin. |
Quick start
Find a directory to save. Must be a simple directory listing, example:
Create a torrent with defaults (auto piece size, curated trackers, hybrid v1+v2):
torrent-ranger https://example.org/big-dataset/
Choose a specific piece size (e.g., 16 MiB) and bump concurrency:
torrent-ranger https://example.org/big-dataset/ -p 16 -c 32
Add your own trackers:
torrent-ranger https://example.org/big-dataset/
-t https://tracker.opentrackr.org/announce https://tracker.openbittorrent.com/announce
Resume an interrupted hashing run (re-uses existing hashes.bin; still builds torrent at the end):
torrent-ranger https://example.org/big-dataset/ --resume
Use a pre-existing manifest and hashes (build only):
torrent-ranger https://example.org/big-dataset/ --skip-snapshot --skip-hash
Change the output location:
torrent-ranger https://example.org/big-dataset/ -o /mnt/archives/datasets
Produce v1-only (no v2/hybrid):
torrent-ranger https://example.org/big-dataset/ --v1-only
Troubleshooting
- "Manifest file not found" — you used --skip-snapshot but manifest.jsonl isn't present under the dataset folder.
- "Hashes file not found" — you used --skip-hash but hashes.bin is missing.
- "Hash list has N entries but layout needs M ... rerun with --resume." — piece layout changed or hashing was interrupted; resume with consistent flags.