seo-agent

A local SEO co-pilot built with Python, Browser Use, and the Claude API. Visits real pages in a visible browser window, extracts SEO signals, checks for broken links, scores backlinks, surfaces GSC quick wins, maps internal link clusters, and writes structured reports — resumable if interrupted.

seo-agent visiting a page in a real Chromium browser

Ran it on my own sites. Found a title cannibalising its own homepage, a position 9.5 query with 0% CTR, two missing internal links, and an orphan page with no path to it.

Everything is open source.

• What It Does
• Stack
• Installation
• Configuration
• Usage
• Modules
• Output
• PASS/FAIL Rules
• Cost
• Scheduling
• Attestation Verification
• Environment Variables
• Architecture
• Contributing
• Writing
• License
• Limitations

Core audit (runs on a URL list):

• Visits each URL in a real Chromium browser — not a headless scraper
• Extracts title, meta description, H1s, and canonical tag via Claude API
• Checks for broken same-domain links asynchronously using httpx
• Detects edge cases (404s, login walls, redirects) and pauses for human input
• Writes results to report.json incrementally — safe to interrupt and resume
• Generates a plain-English report-summary.txt on completion

Standalone modules (run independently on any data):

• qualify-backlinks — score a list of referring domains for niche relevance and traffic quality
• gsc-insights — parse a Search Console export and find quick wins and cannibalisation
• relevance-score — score candidate pages as internal link sources for a target URL
• cluster-audit — map your full site into topic clusters, find orphans and missing hubs
• serp-features — query SearchApi for each target query and detect which SERP features are present (AI Overview, featured snippet, PAA, image pack, video results, local pack, knowledge panel)
• llm-visibility — query Claude, ChatGPT, Gemini, Perplexity, and Bing Copilot with your target queries and check whether your domain gets cited across each engine

• Browser Use — real browser navigation via Playwright
• Anthropic Claude API — structured SEO signal extraction (Haiku for modules, Sonnet for core)
• SearchApi — structured Google SERP data for feature detection, plus ChatGPT, Gemini, Perplexity, and Bing Copilot answer-engine queries for LLM visibility
• httpx — async broken link detection and SearchApi calls
• Python 3.11+, flat JSON state files, no database required

git clone https://github.com/dannwaneri/seo-agent
cd seo-agent
pip install -r requirements.txt
playwright install chromium

Set your Anthropic API key:

# macOS/Linux
export ANTHROPIC_API_KEY="sk-ant-..."
# Windows PowerShell
$env:ANTHROPIC_API_KEY = "sk-ant-..."

Or add it to a .env file in the project root — the agent loads it automatically:

ANTHROPIC_API_KEY=sk-ant-...

Add your URLs to input.csv:

url
https://example.com
https://example.com/about
https://example.com/contact

Interactive mode — pauses on edge cases (login walls, 404s):

python main.py

Auto mode — skips edge cases, logs them to needs_human[] in state, continues:

python main.py --auto

Resume after interruption — already-audited URLs are skipped automatically:

python main.py --auto
# Starting audit: 4 pending, 3 already done.

With project isolation (each client gets separate input, state, and reports):

python main.py --project acme --auto
python main.py --project globex --auto

With cost-curve tiered routing (cheaper checks first):

python main.py --tiered --auto

With AI rewrite suggestions:

python main.py --rewrite --auto
python main.py --rewrite --voice-sample my-writing-sample.txt --auto

With viewport screenshots (saves one PNG per URL to screenshots/):

python main.py --tiered --screenshot
# screenshots/dannwaneri-com.png
# screenshots/dannwaneri-com-ai-agents.png

Backlink qualifier — score a .txt or .csv list of referring domain URLs:

python main.py qualify-backlinks backlinks.txt --niche "AI agents python"
python main.py qualify-backlinks backlinks.txt --niche "SEO tools" --project acme

GSC insights — parse a Search Console query export:

python main.py gsc-insights gsc-export.csv
python main.py gsc-insights gsc-export.csv --min-impressions 100 --project acme

Relevance scorer — score candidate pages as internal link sources:

python main.py relevance-score --target https://example.com/target-page --pages pages.txt
python main.py relevance-score --target https://example.com/target --pages pages.txt --project acme

Cluster audit — map site pages into topic clusters:

python main.py cluster-audit --pages pages.txt
python main.py cluster-audit --pages pages.txt --project acme

SERP feature detection — detect which SERP features are present for your target queries (via SearchApi):

# single query
python main.py serp-features --query "does twitch pay nigerians" --project naija-payments
# from a GSC export (checks top 20 queries by impressions)
python main.py serp-features --queries gsc-export.csv --project naija-payments
# plain text file, one query per line
python main.py serp-features --queries queries.txt --max 10 --project acme

Requires a SearchApi key (see Environment Variables):

export SEARCHAPI_KEY="your-key-here"

Outputs serp-features.md — a feature matrix table and per-query opportunity notes.

LLM visibility checker — check whether Claude, ChatGPT, Gemini, Perplexity, and Bing Copilot cite your domain when answering your target queries:

# against a plain text file of queries, all five engines
python main.py llm-visibility --domain dannwaneri.com --queries queries.txt --project dannwaneri-com
# against a GSC export (checks top 20 queries by impressions), specific engines only
python main.py llm-visibility --domain naija-vpn.com --queries gsc-export.csv --project naija-payments --engines chatgpt,gemini

Requires ANTHROPIC_API_KEY for the Claude check and SEARCHAPI_KEY for the other four (see Environment Variables).

Outputs llm-visibility.md — a per-engine score, a citation matrix (query ×ばつ engine), and mention details.

Limitations: Claude's check runs against its training data, so anything published after its cutoff won't appear regardless of quality. The other four engines are queried live and reflect the current web. Re-run quarterly. A 0% score on a new site is expected across all engines; it reflects data availability, not content quality. In practice, don't expect all five engines to agree — see the SearchApi Developer Ambassador piece for real numbers on how little overlap there actually is.

Most modules use Claude Haiku. Prompts are in prompts/. llm-visibility's non-Claude engines go through SearchApi instead. Results write to markdown files in the project directory or repo root.

Scores each referring domain on three axes:

Composite score = niche * 0.50 + traffic * 0.30 - spam * 0.20

Tiers: Insert Worthy (≥80), Useful (60–79), Neutral (40–59), Borderline (20–39), Avoid (<20 or spam >70).

Fetches pages via real browser. Caches results to flat JSON — crash at URL 47, restart at URL 48.

Parses a Search Console query export CSV. Flags quick wins: position 4–20, impressions ≥50 (configurable), CTR <5%. Sends the top 50 rows to Haiku with a prompt that specifically asks for queries where two pages compete — the cannibalisation signal you can't see by sorting a spreadsheet.

Scores candidate pages as sources for internal links pointing at a target URL:

Checks existing links deterministically before scoring — never recommends a link that already exists.

Tiers: Strong Link (≥75), Good Opportunity (55–74), Possible (35–54), Skip (<35).

Builds the full internal link graph from a page list. Counts incoming links per page — zero incoming means orphan. Sends the complete graph to Haiku: cluster mapping, missing hub detection, cross-cluster link suggestions, and a prioritised fix list.

Calls SearchApi's Google engine for each query and detects seven feature types from the structured JSON response — no browser, no CAPTCHA, no bot detection:

Results write to serp-features.md with a feature matrix and per-query opportunities.

Requires a SearchApi account (searchapi.io). Set SEARCHAPI_KEY in your environment before running — the same key used by llm-visibility's non-Claude engines.

Queries each of your target queries against Claude, ChatGPT, Gemini, Perplexity, and Bing Copilot, and checks whether your domain gets cited in the response. Claude runs through the Anthropic API directly against its training data; the other four run live through SearchApi's answer-engine endpoints (modules/searchapi_client.py). Use --engines to run a subset instead of all five.

Results write to llm-visibility.md with a per-engine visibility score, a citation matrix (which engines cited you for which query), and per-engine mention details with the source link that got matched.

Limitations: Claude's check reflects its training data cutoff — content published after that won't appear regardless of quality. The SearchApi-backed engines are queried live, so they reflect the current web, but individual calls can fail (seen both a 503 and a request timeout from Bing Copilot in testing) — a failed query is logged as an error for that engine and doesn't stop the run. A 0% score on a new domain is expected across all engines. In practice, the five engines rarely agree with each other — running only one gives a partial, sometimes misleading picture of your actual AI visibility.

report.json — structured audit result per URL:

{
 "url": "https://example.com",
 "title": { "value": "Example", "length": 7, "status": "PASS" },
 "description": { "value": null, "length": 0, "status": "FAIL" },
 "h1": { "count": 1, "value": "Welcome", "status": "PASS" },
 "canonical": { "value": "https://example.com", "status": "PASS" },
 "broken_links": { "count": 0, "status": "PASS" },
 "flags": ["Meta description is missing"]
}

report-summary.txt — plain-English summary:

https://example.com | FAIL [description]
https://example.com/about | PASS
https://example.com/contact | FAIL [title, canonical]
1/3 URLs passed

Module outputs write to markdown files: backlink-report.md, gsc-insights-report.md, relevance-report.md, cluster-report.md.

The 60-character title limit is a display threshold, not a ranking penalty. Titles over 60 characters get truncated in Google search results. The agent flags display risk, not a ranking violation.

The default audit routes every URL through Claude Sonnet. --tiered routes cheaper checks first.

Without --tiered, every URL uses Tier 3. With --tiered, Tier 1 runs first and escalates only when needed. A 20-URL audit at Tier 3 costs about 0ドル.12.

Module runs (backlink qualifier, GSC insights, etc.) all use Haiku — cost is negligible for typical site sizes.

For weekly audits, schedule a batch file or cron job.

Windows (run-audit.bat):

@echo off
set ANTHROPIC_API_KEY=your-key-here
cd /d C:\path\to\seo-agent
python main.py --auto

macOS/Linux (cron):

# Every Monday at 7am
0 7 * * 1 cd /path/to/seo-agent && ANTHROPIC_API_KEY=your-key python main.py --auto

seo-agent ships with opt-in instrumentation that cross-checks each Claude API call against Anthropic's billing-side usage records — independent verification that catches ghost calls, model substitution, and token-count drift. The pattern lives in production-safe-agent-loop; seo-agent is the first real workload wired into it.

Opt-in. If ATTESTATION_LEDGER_DB is unset (default), every instrumentation point is a no-op. seo-agent runs identically — same behavior, same outputs, no SQLite writes, no overhead. Set the env vars only if you want shadow-mode verification.

How it works:

• Each of the six Claude-calling modules records a fingerprint (timestamp, model, tokens in/out) after a successful messages.create(). Fingerprints land in a local SQLite database.
• run_verifier.py compares fingerprints against Anthropic's usage records, in one of two modes:
  • Admin API mode — hits /v1/organizations/usage_report/messages automatically (Team/Enterprise orgs only).
  • CSV mode — reads an exported CSV from Console > Analytics > Usage (works on individual orgs).
• Drift findings get written to an append-only diff_reports table for review.

Safety guarantees — three independent failsafes:

1. Every record_fingerprint() call is wrapped in try/except: pass. Instrumentation cannot break the agent.
2. Missing production-safe-agent-loop on the path → silent no-op.
3. Unset ATTESTATION_LEDGER_DB → silent no-op.

See core/attestation_setup.py for the helper module and run_verifier.py for the cron script. Configuration is documented in .env.example.

seo-agent/
├── main.py # Entry point — core audit + module sub-commands
├── config.py # SMTP and PageSpeed config helpers
├── input.csv # Default URL list
├── requirements.txt
├── .gitignore
│
├── core/ # Audit engine
│ ├── browser.py # Playwright browser driver
│ ├── extractor.py # Claude API extraction layer
│ ├── linkchecker.py # Async broken link checker
│ ├── hitl.py # Human-in-the-loop pause logic
│ ├── reporter.py # Report writer (JSON + summary)
│ ├── state.py # State persistence + run history
│ └── index.py # Standalone core entry point
│
├── modules/ # Standalone analysis modules
│ ├── backlink_qualifier.py
│ ├── cluster_audit.py
│ ├── gsc_insights.py
│ ├── relevance_scorer.py
│ ├── serp_features.py # SearchApi-based SERP feature detection
│ ├── llm_visibility.py # Multi-engine LLM visibility checker (Claude + SearchApi engines)
│ └── searchapi_client.py # SearchApi.io client — ChatGPT, Gemini, Perplexity, Bing Copilot
│
├── prompts/ # Haiku prompt templates for each module
│ ├── backlink_qualifier.md
│ ├── cluster_audit.md
│ ├── gsc_insights.md
│ └── relevance_scorer.md
│
└── premium/ # Optional paid features (PDF reports, email delivery)
 ├── cost_curve.py
 ├── enhanced_reporter.py
 ├── rewrite_agent.py
 ├── pagespeed.py
 ├── structured_data.py
 └── email_reporter.py

Pull requests are welcome. The full codebase — core audit engine, all four modules, all prompts — is open source.

To contribute:

git clone https://github.com/dannwaneri/seo-agent
cd seo-agent
pip install -r requirements.txt
playwright install chromium
# make your changes
# run the inline acceptance tests
python main.py # runs __main__ test block

Articles about building and running this agent:

• How to Build a Local SEO Audit Agent with Browser Use and Claude API — freeCodeCamp
• I Ran My Own SEO Agent on My Two Domains — It Went from 0.4 to 44 in One Afternoon — dev.to
• I Was Paying 0ドル.006 per URL for SEO Audits Until I Realized Most Needed 0ドル — dev.to
• How to Build a Cost-Efficient AI Agent with Tiered Model Routing — freeCodeCamp
• I Built a Local AI Agent That Audits My Own Articles — It Flagged Every Single One — dev.to
• I Gave My SEO Agent a Real Site. It Found Bugs I'd Missed for Weeks. — the co-pilot build with all 4 modules
• I Tested 5 AI Engines On My Own Sites. None Agreed. — dev.to, the SearchApi multi-engine llm-visibility build

Known constraints to be aware of before running:

Core audit

• Runs headfully (visible browser window) — not designed for server-side CI pipelines
• Per-URL delay of 2 seconds is intentional; removing it may cause rate-limiting on target servers
• Broken link checker only checks same-domain links by default

SERP feature detection (serp-features)

• Requires a SearchApi key (SEARCHAPI_KEY), shared with llm-visibility's non-Claude engines.
• Results reflect the SERP at time of the API call, not a guaranteed match for what you'd see from your exact location.
• answer_box (featured snippet) rarely fires on live results now — AI Overview has replaced it for most query types tested so far.
• AI Overview detection checks the ai_overview top-level key and related_questions type — if Google changes how it structures this in the response, the detection may need updating.

LLM visibility (llm-visibility)

• Tests Claude (Haiku model, via training data) plus ChatGPT, Gemini, Perplexity, and Bing Copilot (live, via SearchApi) — pass --engines to limit to a subset
• Claude's result reflects a training data cutoff; recently published content will not appear regardless of quality
• A visibility score of 0% on a site less than 1–2 years old is normal and expected across all five engines
• The SearchApi-backed engines reflect the current web, not a training snapshot — re-run Claude's check quarterly; the others can be re-run any time
• ChatGPT only returns cited sources when web_search=true is set — the client sets this automatically, but it's easy to miss if you're calling SearchApi directly
• Bing Copilot has occasionally errored (503, timeout) during testing — a failed engine query is logged and skipped, it doesn't abort the run
• Requires SEARCHAPI_KEY for the four SearchApi-backed engines; Claude only needs ANTHROPIC_API_KEY

Backlink qualifier

• Scores are Claude's assessment based on page content at time of fetch — not Ahrefs/Moz DR scores
• Cache (backlink-state.json) is never invalidated automatically; stale entries accumulate over time

Daniel Nwaneri — dannwaneri.com · DEV.to · GitHub

MIT. See LICENSE.