anubis-fetch

Built with Nix CI License: MIT

$ nix run github:fzakaria/anubis-fetch -- https://lore.kernel.org/linux-mm/some-thread/T/

A fast (Go) CLI to fetch URLs from behind Anubis proof-of-work walls and Cloudflare fingerprint checks — solving the challenge in-process, and only reaching for a real browser when it has to.

A growing number of sites — lore.kernel.org, GNOME, kernel.org, and many more — sit behind Anubis, a bot-wall that makes your browser solve a SHA-256 proof-of-work before it serves any content. It's great at stopping scrapers. It's also great at stopping you when you just want to curl a mailing-list thread:

$ curl -s https://lore.kernel.org/linux-mm/some-thread/T/ | grep -o '<title>.*</title>'
<title>Making sure you&#39;re not a bot!</title> # 🤢

anubis-fetch gets you the actual page:

$ anubis-fetch https://lore.kernel.org/linux-mm/some-thread/T/ | grep -o '<title>.*</title>'
<title>[PATCH 0/2] ...</title> # 🥳

Bot-walls live at two different layers, and anubis-fetch handles both, cheapest step first:

1. Reuse a saved cookie. Anubis hands out a signed auth cookie (techaro.lol-anubis-auth) after you pass once; a browser isn't re-challenged on the next visit, and neither are we. Cookies are persisted per host (see Cookie persistence).
2. Solve the proof-of-work in-process. The request goes out over req impersonating a real Chrome — same TLS/JA3 + HTTP/2 fingerprint — which also clears Cloudflare's passive fingerprinting. If the response is an Anubis challenge, we brute-force the nonce and submit it. No browser, ~0.6s.
3. Fall back to a real browser. For anything the fast path can't do, we drive headless Chromium via chromedp, which runs whatever JavaScript the site serves.

Because it's ×ばつ slower and drags a ~200 MB Chromium into every fetch. The in-process solver is the common case; the browser is the safety net.

Anubis embeds a challenge as JSON in the page:

{"rules":{"algorithm":"fast","difficulty":4},
 "challenge":{"id":"...","method":"fast","randomData":"6214bd88...","difficulty":4}}

Solving it means finding a nonce such that hex(sha256(randomData ‖ nonce)) begins with difficulty zero characters (the nonce is its base-10 string). We then hand the answer back:

GET /.within.website/x/cmd/anubis/api/pass-challenge?id=...&response=<hash>&nonce=<n>&redir=<url>&elapsedTime=<ms>

...which sets the auth cookie and redirects to the real page. Difficulty 4 (the default on lore/kernel.org/GNOME) is ~65k hashes — sub-millisecond in Go.

Run it directly:

$ nix run github:fzakaria/anubis-fetch -- <url>

Install into your profile:

$ nix profile install github:fzakaria/anubis-fetch

Or add it to your own flake:

{
 inputs.anubis-fetch.url = "github:fzakaria/anubis-fetch";
 # then, e.g. in home.packages / environment.systemPackages:
 # inputs.anubis-fetch.packages.${system}.default
}

$ anubis-fetch [flags] URL

# HTML to stdout
$ anubis-fetch https://lore.kernel.org/linux-mm/some-thread/T/
# readable plain text
$ anubis-fetch --text https://lore.kernel.org/linux-mm/some-thread/T/
# lean/fast only — useful in scripts; exits 3 if it would need a browser
$ anubis-fetch --no-browser https://example.com/ && echo "got it"

After a successful fetch, the auth cookie is written to $XDG_CACHE_HOME/anubis-fetch/cookies/<host>.json (falling back to ~/.cache/...). The next run for that host is let straight through — no proof-of-work, no browser — exactly like a browser revisit. Cookies obtained via the browser fallback are saved too, so a subsequent run can take the fast HTTP path. Use --no-cache to disable, or just delete the file to force a re-solve.

Everything is wired through the flake:

$ nix develop # dev shell: go, gopls, chromium, treefmt
$ go test ./... # unit tests (hermetic — no network)
$ nix build # build the wrapped binary
$ nix flake check # build + tests + formatting
$ nix fmt # format Go + Nix via treefmt (gofmt + alejandra)

The proof-of-work implementation is pinned against Anubis' own published test vector (sha256("hunter" + "0")) so a drift in the hash construction fails a unit test rather than silently returning garbage.

• TecharoHQ/anubis — the bot-wall this politely negotiates with.
• imroc/req — the HTTP client whose Chrome impersonation clears passive Cloudflare fingerprinting.
• chromedp/chromedp — drives the headless-browser fallback.