内容
<p align="center">
<img src="assets/banner.svg" alt="screamingfrog-audit-mcp — drive the Screaming Frog SEO Spider from any MCP client" width="100%">
</p>
<p align="center">
<a href="https://github.com/mshahiddigital/screamingfrog-audit-mcp/actions/workflows/ci.yml"><img src="https://github.com/mshahiddigital/screamingfrog-audit-mcp/actions/workflows/ci.yml/badge.svg" alt="CI"></a>
<a href="https://pypi.org/project/screamingfrog-audit-mcp/"><img src="https://img.shields.io/pypi/v/screamingfrog-audit-mcp" alt="PyPI"></a>
<a href="https://www.python.org/"><img src="https://img.shields.io/badge/python-3.10%2B-blue" alt="Python 3.10+"></a>
<a href="LICENSE"><img src="https://img.shields.io/badge/license-MIT-green" alt="License: MIT"></a>
</p>
Drive the [Screaming Frog SEO Spider](https://www.screamingfrog.co.uk/seo-spider/)
from Claude, Cursor, or any other MCP client. Crawl a site, get a ranked issue
register back, ask questions of the crawl data, and render a shareable report —
without opening the GUI or writing a single command.
### It works on the free, unlicensed SEO Spider
That is the point of this server, and it is unusual: the other MCP servers for
Screaming Frog build on its saved-crawl database, which is a licensed feature,
so they need a paid install. This one drives the crawl directly and never
touches that database.
The free tier caps you at 500 URLs **per invocation** — not per site. So
`full=true` reads `robots.txt` and the sitemaps, splits the URLs into batches
under the cap, runs each through list mode, and merges the exports. A
3,000-page site audits completely on a free install.
A licence removes the cap and unlocks `config=` for JavaScript rendering and
custom extraction. Both tiers are supported and the server adapts to whichever
you have.
```
You: Crawl example.com and tell me what's actually broken.
→ start_crawl(url="https://example.com")
→ crawl_status() # 248 URLs, 51s
→ get_issues(priority="High")
Claude: Three high-priority problems. The big one: robots.txt disallows
/_next/, which hides 85 JS and CSS bundles from Google...
```
## Install
### Step 1 — the two things you need
**Screaming Frog SEO Spider**, installed on the same computer
([download](https://www.screamingfrog.co.uk/seo-spider/)). The free version is
fine. **Python 3.10 or newer** — macOS and most Linux systems already have it;
on Windows get it from [python.org](https://www.python.org/downloads/) and tick
**"Add python.exe to PATH"** on the first screen.
Then install `uv`, which runs the server for you. Open a terminal and paste one
line:
**macOS / Linux** — open *Terminal*:
```bash
curl -LsSf https://astral.sh/uv/install.sh | sh
```
**Windows** — open *PowerShell* (press Start, type `powershell`):
```powershell
powershell -ExecutionPolicy ByPass -c "irm https://astral.sh/uv/install.ps1 | iex"
```
**Close the terminal and open a new one** so it picks up the change.
### Step 2 — one command sets up your apps
```bash
uvx screamingfrog-audit-mcp --install
```
That finds Claude Desktop, Cursor and VS Code on your machine and configures
each one. You never open a JSON file. It:
- **keeps everything already in your config** — other MCP servers and settings
are untouched
- **writes the full path** to the program, because Claude Desktop and other
desktop apps cannot find commands the way a terminal can
- **saves a timestamped backup** before changing anything
- **re-reads the file afterwards** to prove it is still valid, and restores your
backup if it is not
- **refuses to touch a config that is already broken**, and tells you instead
Want to see what it would do first? Add `--dry-run`. Want one app only?
`--install claude-desktop`, `--install cursor` or `--install vscode`.
Then **fully quit and reopen the app** — on macOS that is Cmd+Q, not just
closing the window. Ask it: *"check my screaming frog install"*.
<details>
<summary><b>Only using Claude Desktop? Full walkthrough, Mac and Windows</b></summary>
You still need a terminal **once**, to install the pieces. After that you never
open one again — everything happens inside Claude Desktop.
**1. Install Screaming Frog SEO Spider**
[Download it](https://www.screamingfrog.co.uk/seo-spider/) and open it once so
it finishes setting itself up. The free version is fine. Then close it.
**2. Install Python** (skip on macOS, it is already there)
Windows: [python.org/downloads](https://www.python.org/downloads/), run the
installer, and **tick "Add python.exe to PATH"** on the first screen. That
checkbox matters; without it nothing below works.
**3. Open a terminal**
- **macOS** — press Cmd+Space, type `Terminal`, press Enter
- **Windows** — press the Start key, type `powershell`, press Enter
**4. Install uv** — copy the line for your system, paste, press Enter
macOS:
```bash
curl -LsSf https://astral.sh/uv/install.sh | sh
```
Windows:
```powershell
powershell -ExecutionPolicy ByPass -c "irm https://astral.sh/uv/install.ps1 | iex"
```
**5. Close that terminal window and open a new one.** It needs a fresh start to
see what you just installed.
**6. Set up Claude Desktop** — paste this and press Enter:
```bash
uvx screamingfrog-audit-mcp --install claude-desktop
```
You should see `[ OK ] Claude Desktop: Added to ...`. It writes the settings
file for you and backs up anything already there.
**7. Fully quit Claude Desktop and reopen it**
- **macOS** — Cmd+Q. Closing the window is not enough
- **Windows** — right-click the Claude icon near the clock, choose Quit
**8. Check it worked.** In Claude Desktop, ask:
> check my screaming frog install
It should report the Spider it found and your licence tier. Then try:
> crawl example.com and tell me what is broken
**If step 6 says "command not found"**, step 5 was skipped — close the terminal
and open a new one. **If Claude Desktop shows no tools**, it was not fully
quit; use Cmd+Q or the tray icon, not the window close button.
</details>
### Command-line assistants
These have their own one-liners, so there is nothing to edit:
```bash
claude mcp add screaming-frog -- uvx screamingfrog-audit-mcp
codex mcp add screaming-frog -- uvx screamingfrog-audit-mcp
hermes mcp add screaming-frog --command uvx --args screamingfrog-audit-mcp
```
### If something is not working
```bash
uvx screamingfrog-audit-mcp --doctor
```
It checks Python, the MCP SDK, whether the Spider is found *and actually runs*,
which options your Screaming Frog version supports, your licence tier, and
whether the audit folder is writable — then prints the exact config for your
machine.
```
[PASS] Python: Python 3.12.7 on Darwin
[PASS] MCP SDK: MCP SDK 2.1.1, using MCPServer (mcp 2.x)
[FAIL] SEO Spider: Screaming Frog SEO Spider not found
Install it from https://www.screamingfrog.co.uk/seo-spider/ ,
or set SCREAMING_FROG_PATH to the executable.
```
An MCP server talks over a pipe, so when it fails to start the app just shows a
dead server with no reason. The doctor is how you see the reason.
<details>
<summary><b>Editing the config by hand instead</b></summary>
`--install` exists so you do not have to, but if you prefer:
| App | File |
|---|---|
| Claude Desktop (macOS) | `~/Library/Application Support/Claude/claude_desktop_config.json` |
| Claude Desktop (Windows) | `%APPDATA%\Claude\claude_desktop_config.json` |
| Cursor | `~/.cursor/mcp.json` |
| VS Code | Command Palette → *MCP: Open User Configuration* |
Three things go wrong by hand, all silently:
1. **Use the full path to `uvx`, never bare `uvx`.** Desktop apps launch
without your shell's `PATH`. Find it with `which uvx` (macOS/Linux) or
`where.exe uvx` (Windows).
2. **On Windows, double every backslash.** `C:\Users\you\...` must be written
`C:\\Users\\you\\...` in JSON, or the file is invalid and the app ignores
*every* server in it.
3. **Add to what is already there.** Do not paste over the file.
```json
{
"mcpServers": {
"screaming-frog": {
"command": "/Users/you/.local/bin/uvx",
"args": ["screamingfrog-audit-mcp"]
}
}
}
```
VS Code differs: the key is `servers`, not `mcpServers`, and each entry needs
`"type": "stdio"`.
</details>
### ChatGPT, and what cannot work
The **ChatGPT desktop app** bundles Codex and reads the same
`~/.codex/config.toml`, so the `codex mcp add` line above configures it too.
**ChatGPT on the web or on your phone cannot use this**, and no config changes
that. Those connectors call a remote web address, while this is a program that
runs on your computer and drives the Screaming Frog installed there. It has to
live where the Spider lives. The same is true of any assistant that only
accepts a remote URL.
### Optional settings
Add these to the `env` block of any config (a `[mcp_servers.screaming-frog.env]`
table in Codex):
| Variable | What it does |
|---|---|
| `SCREAMING_FROG_PATH` | Path to the SEO Spider executable, if it is installed somewhere non-standard |
| `SF_MCP_AUDIT_DIR` | Where crawl folders are written (default `~/.screamingfrog-audit-mcp/audits`) |
| `SF_ALLOWED_DOMAINS` | Comma-separated domains this server may crawl. Set it for anything unattended |
```json
{
"mcpServers": {
"screaming-frog": {
"command": "uvx",
"args": ["screamingfrog-audit-mcp"],
"env": {
"SF_MCP_AUDIT_DIR": "/Users/you/audits",
"SF_ALLOWED_DOMAINS": "example.com,acme.co.uk"
}
}
}
}
```
## The free-tier situation
The received wisdom is that the Screaming Frog CLI needs a licence. It does
not. Verified against a build reporting `Licence Status: Missing`:
| | |
|---|---|
| **Works unlicensed** | `--headless`, spider / list / sitemap crawl modes, every tab export, every saved report, sitemap generation |
| **Licence-gated** | save/load crawl, crawl comparison, config files, JavaScript rendering, scheduling, and the GA4 / Search Console / PageSpeed / Ahrefs / Moz integrations |
| **Capped** | 500 URLs **per invocation** — not per site |
Because the cap is per invocation, `start_crawl(full=true)` discovers URLs from
`robots.txt` and the sitemaps, batches them under the cap through list mode,
and merges the exports back into one set. That crawls a site of any size on the
free tier.
A licence removes the cap and makes `full` unnecessary. Everything else works
the same.
## Tools
| Tool | What it does |
|---|---|
| `check_install` | Install status, licence status, current limits, and how to fix a failed lookup |
| `available_filters` | The export names **your** Screaming Frog build accepts |
| `start_crawl` | Background headless crawl. `full` beats the free cap, `everything` exports every table |
| `crawl_status` | Poll a running crawl. Omit `job_id` for the most recent |
| `cancel_crawl` | Stop a crawl, keep partial exports |
| `list_crawls` | Crawl folders, newest first, with headline counts |
| `get_issues` | The priority-ranked issue register. **Start here** |
| `get_analysis` | What the *set* of URLs means: depth, link equity, sitemap accuracy, content depth, performance, indexability, duplication |
| `list_exports` | The CSV exports in a crawl, with row counts |
| `read_export` | Rows from one export: column-selectable, paged, capped, filtered by `contains` / `exact` / `regex` on any column |
| `aggregate_export` | **Counts and group-by without returning rows** — "how many 404s", "status codes by folder" — in one small response |
| `storage_summary` | Disk used per saved crawl, largest first |
| `delete_crawl` | Permanently delete a crawl folder (requires `confirm`) |
| `build_report` | The branded deliverable: **`audit-workbook.xlsx`** (Summary, Issue register, Analysis, Data index, and every export as its own highlighted sheet), a printable `report.html`, `report.md` and `analysis.json`. `consolidate=True` folds the CSV exports into the workbook and deletes them, leaving one file |
## Two design decisions worth knowing
**Crawls are background jobs.** A crawl takes minutes; an MCP call should
answer in seconds. `start_crawl` forks a detached child and hands back a
`job_id`. Nothing blocks unless you pass `wait_seconds`. The crawl survives the
MCP server restarting.
**Reads are capped, on purpose.** A finished crawl folder is tens of megabytes
of CSV. Feeding that to a model is both useless and expensive. Every read tool
caps at 500 rows, lets you pick columns, and truncates long cells. Ask
`get_issues` first — it's the whole site in about 60 lines — and
`aggregate_export` when the question is "how many" or "broken down by", since
counting rows by hand through a model is the expensive way to get a number.
Reach for `read_export` only when you actually need the rows.
## The deliverable
`build_report` writes four files into the crawl folder:
| File | What it is |
|---|---|
| `audit-workbook.xlsx` | The master workbook. Summary, Issue register, Analysis, Data index, and **every crawl export as its own sheet** — around 70 tables on a full crawl |
| `report.html` | Printable summary. Open it and Print to PDF |
| `report.md` | The same content as plain text |
| `analysis.json` | The derived layer, machine-readable |
Every sheet has a frozen header, autofilter, banded rows, sized columns and a
coloured tab. **Cells are highlighted where the value is the finding** — issue
priority, 4xx/5xx status codes, non-indexable URLs, thin content, slow
responses — so the problems are visible before you read a cell.
Reports carry a credit line naming the tool and its author.
### One workbook instead of seventy CSVs
A finished crawl leaves around seventy exports in the folder, and the workbook
already carries every one of them. `build_report(consolidate=True)` folds them
in and deletes them, so the folder holds the workbook, the report files, and a
`consolidated.json` manifest naming which sheet holds which table.
The delete is earned. Each sheet is written, the saved workbook is **reopened
from disk**, and every sheet is checked against the row count it should hold.
Only then is a file removed. If the save fails, the reopen fails, or a sheet
comes back short, nothing is deleted and the reason is reported. An export too
large to carry in full is kept on disk and named in the manifest, because a
sampled sheet is not a substitute for the file.
It is off by default: `read_export` and `aggregate_export` read those CSVs. Turn
it on when the folder is a finished deliverable rather than a crawl you are
still asking questions about. Those tools then explain the consolidation and
point at the workbook instead of reporting the folder as empty.
## Where crawls are stored
`~/.screamingfrog-audit-mcp/audits/<label>/` by default. Each folder holds the raw
Screaming Frog CSV exports, `audit-summary.json`, and whatever
`build_report` wrote — or, after a consolidated report, the workbook and
manifest in place of those exports.
Override with `SF_MCP_AUDIT_DIR`:
```json
{
"mcpServers": {
"screaming-frog": {
"command": "uvx",
"args": ["screamingfrog-audit-mcp"],
"env": { "SF_MCP_AUDIT_DIR": "/Users/you/audits" }
}
}
}
```
Set `SCREAMING_FROG_PATH` if the Spider is installed somewhere non-standard.
## Restricting what it may crawl
By default this server will crawl any host it is asked to. That is fine on your
own machine, and a liability when an agent runs unattended: a confused or
prompt-injected one can point a crawler at internal addresses or at third
parties who did not ask to be crawled.
Set `SF_ALLOWED_DOMAINS` and `start_crawl` refuses anything else:
```json
"env": { "SF_ALLOWED_DOMAINS": "example.com,acme.co.uk" }
```
Subdomains of a listed domain are allowed; look-alikes are not, so
`shop.example.com` passes and `example.com.evil.com` does not. `--doctor`
reports whether an allowlist is active.
### Environment variables
All three are listed with examples under
[Optional settings](#optional-settings) in the install section.
## Use it without MCP
The crawl pipeline is a plain module:
```bash
python -m screamingfrog_audit_mcp.runner --url https://example.com --output ./audit
python -m screamingfrog_audit_mcp.runner --url https://example.com --output ./audit --full
```
## Gotchas found the hard way
- **An unknown command-line flag aborts the whole crawl**, it is not ignored:
`FATAL - SeoSpider failed to start ... UnrecognizedOptionException`. The
option set differs between versions — `--skip-empty` does not exist in 19.8,
for instance — so this server reads the binary's own `--help` and passes only
what your build advertises. Works on old and new versions alike.
- **One wrong filter name aborts the whole crawl.** Screaming Frog renames tab
filters between versions, and an unrecognised name fails the run with a Java
stack trace rather than skipping it. Every name is validated against your
installed binary at crawl time, so an upgrade degrades instead of breaking.
- **Its own `--help` output contains a poisoned entry.** The binary lists a
placeholder `UNDEF:Unknown`, and passing it back aborts the crawl with
`Using UNDEF as tab is not supported`. It's filtered out.
- **`os.kill(pid, 0)` is not a liveness probe on Windows.** Any signal other
than CTRL_C/CTRL_BREAK routes to `TerminateProcess`, so the usual "does this
pid exist" idiom would kill the crawl and then report it finished. Windows
uses `tasklist` to check and `taskkill` to cancel, and never signals.
Detaching differs too: `start_new_session` is POSIX-only.
- **`everything` mode is curated, not literal.** The Spider lists ~1,150 tab
filters, but 800+ are Custom Extraction / Custom Search / Custom JavaScript /
AI filters that need a licence-gated config file and are permanently empty.
Requesting them costs minutes and returns nothing, so those groups plus the
API-dependent ones are excluded.
## Development
```bash
git clone https://github.com/mshahiddigital/screamingfrog-audit-mcp
cd screamingfrog-audit-mcp
pip install -e ".[dev]"
pytest
```
The test suite runs on synthetic export fixtures, so it passes on a machine
that has never had Screaming Frog installed.
## Changelog
See [CHANGELOG.md](CHANGELOG.md).
## License
MIT. Not affiliated with or endorsed by Screaming Frog Ltd. You need your own
copy of the SEO Spider; issue names, descriptions and fix guidance in the
output are Screaming Frog's own.
连接信息
猜你喜欢
暂无推荐服务器