A command-line utility to save web pages from a sitemap to the Web Archive (archive.org).

• Parse standard XML sitemaps
• Save pages to Web Archive with full captures (screenshots, outlinks)
• Filter pages by last modification date
• Concurrent processing for faster archiving

Requires Go 1.22 or later:

go install github.com/iyaki/web-archiver/v2@latest

Download pre-built binaries from the Releases page.

This program requires Web Archive API credentials as environment variables:

• WAYBACK_S3_ACCESS_KEY - Your Web Archive S3 access key
• WAYBACK_S3_SECRET_KEY - Your Web Archive S3 secret key

Get your API keys from Web Archive S3-Like API.

web-archiver <sitemap_url> [<date>]

• sitemap_url (required): URL to the sitemap XML file
• date (optional): Filter date in ISO format (YYYY-MM-DD). Only URLs with lastMod newer than this date will be saved.

Save all URLs from a sitemap:

web-archiver https://iyaki.ar/sitemap.xml

Save only URLs modified since a specific date:

web-archiver https://iyaki.ar/sitemap.xml 2024年05月01日