🌐 SiteClonePro

Most website copiers (like legacy wget, HTTrack, or basic scraper scripts) produce broken offline copies because modern web apps use dynamic CSS url() fonts, CDNs, srcset, lazy-loaded images, and JavaScript loaders that freeze offline. SiteClonePro was built from the ground up to solve all of these problems.

SiteClonePro analyzes the original server's URL pathnames and mirrors them directly on your disk:

• https://example.com/assets/css/main.css ➔ assets/css/main.css
• https://example.com/assets/img/logo/brand.png ➔ assets/img/logo/brand.png
• https://example.com/assets/fonts/font.woff2 ➔ assets/fonts/font.woff2
• https://example.com/products/item-42 ➔ products/item-42/index.html

Every single reference in every HTML and CSS file is converted to point to its corresponding local file based on directory depth:

• In index.html: assets/css/main.css
• In products/index.html: ../assets/css/main.css
• In products/category/item/index.html: ../../../assets/css/main.css
• In assets/css/main.css: ../fonts/font.woff2

• CSS Dependencies: Automatically scans @import url(...), @font-face { src: url(...) }, background-image: url(...), and cursor/mask URLs recursively.
• HTML Tags: Extracts link[href], script[src], img[src], picture, source[srcset], video[src/poster], audio[src], and SVG <use xlink:href>.
• Lazy Load Attributes: Automatically discovers data-src, data-lazy-src, data-original, data-bg, and data-srcset.

• Broken Image Protection: If a remote server returns 404 Not Found for an asset, SiteClonePro provides a transparent placeholder buffer so your offline pages never show broken image icons.
• Offline Preloader Bypass: Modern sites often freeze on loading spinners offline because analytics or tracking scripts are blocked. SiteClonePro injects a micro-script that auto-dismisses preloaders once the DOM is ready.

• Automatically checks /sitemap.xml and /sitemap_index.xml to discover all pages immediately.
• Crawls internal pages concurrently, filtering out external domains, auth routes, and anchor fragments.

• An in-memory cache saves discovered assets after scanning. Clicking Download ZIP reuses the scan results instantly—no redundant crawling!

 ┌─────────────────┐
 │ Input Site URL │ ➔ Validate & emulate modern browser headers
 └────────┬────────┘
 │
 ▼
 ┌─────────────────┐
 │ Sitemap & Crawl │ ➔ Concurrently crawl internal pages & build link graph
 └────────┬────────┘
 │
 ▼
 ┌─────────────────┐
 │ Deep CSS & Font │ ➔ Recursively parse stylesheets for fonts, icons & bg images
 └────────┬────────┘
 │
 ▼
 ┌─────────────────┐
 │ Concurrent Down │ ➔ Download all assets in parallel (14x concurrency + retries)
 └────────┬────────┘
 │
 ▼
 ┌─────────────────┐
 │ Relative Rewrite│ ➔ Convert all URLs to local relative paths (zero remote links)
 └────────┬────────┘
 │
 ▼
 ┌─────────────────┐
 │ ZIP Generation │ ➔ Stream compressed offline-ready ZIP archive
 └─────────────────┘

• Node.js (v18.18 or higher recommended)
• npm, pnpm, or yarn

git clone https://github.com/lobdp/WebMirror.git
cd WebMirror

npm install

npm run dev

Visit http://localhost:3000 in your browser.

npm run build
npm run start

Select the crawl depth that matches your needs:

• Framework: Next.js 16 (App Router)
• Language: TypeScript
• UI & Styling: Vanilla CSS Design System with TailwindCSS v4
• HTML Engine: Cheerio
• Archiving Engine: JSZip

When you unzip the downloaded archive, your website will be ready to open locally:

my-site-offline-site/
├── index.html <-- Main homepage (double-click to open!)
├── about/
│ └── index.html <-- Clean relative subpage
├── products/
│ ├── index.html
│ └── item-1/
│ └── index.html
├── assets/
│ ├── css/
│ │ ├── bootstrap.min.css
│ │ └── main.css
│ ├── js/
│ │ ├── jquery.min.js
│ │ └── main.js
│ ├── img/
│ │ ├── logo/
│ │ └── banner.jpg
│ └── fonts/
│ └── font.woff2
├── external/ <-- External CDN libraries mirrored locally
│ └── code.jquery.com/
├── DOWNLOAD-REPORT.txt <-- Complete audit report of all mirrored assets
└── README.txt <-- Offline instructions

Contributions, issues, and feature requests are welcome!
Feel free to check the issues page.

This project is licensed under the MIT License - see the LICENSE file for details.