The current recommendation on the spec for verifiers is:
Enforce a reasonable size limit when fetching human.json (e.g., 1 MB) to avoid abuse
(1MB should be enough for 10,000+ vouched entries.) The browser extensions enforce this, but in a naive way — if the content-length request header indicates a file size smaller than 1MB the extension will download, check the size, and disregard if bigger than 1MB. But it still downloads the whole file, which would allow a malicious user to crash the extension by serving a huge file with a fake header.
This PR improves the crawling logic by stopping the request once the file has exceeded the request.
The current recommendation on the spec for verifiers is:
> Enforce a reasonable size limit when fetching human.json (e.g., 1 MB) to avoid abuse
(1MB should be enough for 10,000+ vouched entries.) The browser extensions enforce this, but in a naive way — if the `content-length` request header indicates a file size smaller than 1MB the extension will download, check the size, and disregard if bigger than 1MB. But it still downloads the whole file, which would allow a malicious user to crash the extension by serving a huge file with a fake header.
This PR improves the crawling logic by stopping the request once the file has exceeded the request.