mcp-retrieval

<h1 align="center">mcp-retrieval</h1> <p align="center"> <b>An MCP server that gives an LLM three web tools: search, image search, and page scraping — no API keys required.</b> </p> <p align="center"> <a href="LICENSE"><img src="https://img.shields.io/badge/License-MIT-8bc34a?style=for-the-badge" alt="License MIT"></a> <img src="https://img.shields.io/badge/Go-1.25+-00ADD8?style=for-the-badge&logo=go&logoColor=white" alt="Go"> <img src="https://img.shields.io/badge/MCP-Server-6E56CF?style=for-the-badge&logo=anthropic&logoColor=white" alt="MCP"> <img src="https://img.shields.io/badge/Transport-stdio_%7C_http-26A5E4?style=for-the-badge" alt="Transport"> <img src="https://img.shields.io/badge/DuckDuckGo-Search-DE5833?style=for-the-badge&logo=duckduckgo&logoColor=white" alt="DuckDuckGo"> <img src="https://img.shields.io/badge/Bing-Images-008373?style=for-the-badge&logo=microsoftbing&logoColor=white" alt="Bing Images"> <img src="https://img.shields.io/badge/uTLS-Fingerprint-1f6feb?style=for-the-badge" alt="uTLS"> </p> <p align="center"> <a href="#tools">Tools</a> · <a href="#quick-start">Quick start</a> · <a href="#configuration">Configuration</a> · <a href="#retrieval-engine">Retrieval engine</a> · <a href="#architecture">Architecture</a> · <a href="CONTRIBUTING.md">Contributing</a> </p> <!-- TODO: add a demo here once it is recorded. GitHub renders a <video> tag inline when the src points at an uploaded asset URL (https://github.com/user-attachments/assets/...). --> --- ## What it is `mcp-retrieval` is a [Model Context Protocol](https://modelcontextprotocol.io) server written in Go. It exposes web retrieval capabilities to any MCP-compatible client (Claude Desktop, IDE agents, custom LLM apps) as three read-only tools. Under the hood it uses the [`retrieval-go`](https://github.com/free-llms-foundation/retrieval-go) library to search the web and fetch pages, returning results as clean Markdown ready to hand to a model. The library needs **no API keys**: web search goes through DuckDuckGo Lite, image search through Bing Images, and page fetching runs the HTML through a readability extractor before converting it to Markdown. To stay reliable against bot protection it impersonates real browsers at the TLS level and can rotate both browser fingerprints and proxies — see [Retrieval engine](#retrieval-engine). Both transports the MCP SDK supports are available and expose the identical tool set: - **stdio** — the client launches the binary and talks over stdin/stdout (the default, ideal for desktop clients). - **http** — a long-running streamable HTTP server (useful for remote/shared deployments). --- ## Tools | Tool | Description | | :--- | :--- | | **`web_search`** | Runs one or more queries in parallel and returns per-query deduplicated, reranked snippets with links. | | **`web_search_images`** | Runs one or more image queries in parallel and returns per-query deduplicated image results. | | **`web_scrape`** | Downloads one or more pages in parallel and returns the main article text as Markdown. | All three are annotated as **read-only**. Each tool returns a structured JSON payload that matches its output schema; the SDK mirrors the same JSON into the text content block for clients that do not read `structuredContent`. ### `web_search` | Parameter | Type | Default | Notes | | :--- | :--- | :--- | :--- | | `queries` | `[]string` | — | **Required.** Executed in parallel. | | `max_results` | `int` | `5` | Snippets per query, capped at `max_results` config (`20`). | | `timeout_ms` | `int64` | `5000` | Whole-call timeout; clamped to `[min, max]` from config. | | `date` | `string` | — | Freshness filter: `d` (day), `w` (week), `m` (month), `y` (year). | ### `web_search_images` | Parameter | Type | Default | Notes | | :--- | :--- | :--- | :--- | | `queries` | `[]string` | — | **Required.** Executed in parallel. | | `max_images` | `int` | `5` | Images per query, capped at `max_images` config (`10`). | | `timeout_ms` | `int64` | `5000` | Whole-call timeout; clamped to `[min, max]` from config. | | `date` | `string` | — | Freshness filter: `d` / `w` / `m` / `y`. | ### `web_scrape` | Parameter | Type | Default | Notes | | :--- | :--- | :--- | :--- | | `urls` | `[]string` | — | **Required.** Downloaded in parallel. | | `robots_txt` | `bool` | `false` | Respect the page's `robots.txt`. | | `timeout_ms` | `int64` | `5000` | Whole-call timeout; clamped to `[min, max]` from config. | | `remove_links` | `bool` | `false` | Strip Markdown links from the text. | | `max_chars` | `int` | `20000` | Truncate page text to N characters, capped at `max_document_chars` config (`20000`). | > Both `queries`/`urls` lists are capped at `max_queries` (`10`) items per call. Queries must be ≤ 512 characters; URLs ≤ 2048 characters and `http`/`https` only. ### Results and counts Every call fans out across the input list and returns one entry per query/URL, each with its own `status` — `success`, `failed`, or `timeout` — so a partial failure still returns the items that did work. `count` is the number of items actually returned, and it can be **lower than the requested `max_results` / `max_images`**: duplicates within a single query's results are removed before the limit is applied, and the upstream may simply have fewer items to give. A smaller `count` is a normal outcome, not an error. Deduplication is **per query, not across queries**. Each entry is deduplicated on its own, so a link found by two of the queries in the same call appears in both entries — dedupe the union yourself if you need it. ### Errors Request-level failures are returned as a tool result with `isError: true` and a plain-text message, not as a JSON-RPC error — the model reads the message and can correct the call itself. Per-item failures never do this; they stay inside the payload as `status: "failed"` / `"timeout"`. A call fails outright only when the input is rejected before any work starts, or when **every** item in it fails: | Message | Meaning | | :--- | :--- | | `invalid request` | The arguments did not pass validation. | | `too many queries` / `too many urls` | The list exceeds `MAX_QUERIES`. | | `query must not be empty` | An empty query, or an empty `queries` list. | | `query is too long` | A query exceeds 512 characters. | | `invalid url` | A URL is malformed, over 2048 characters, or not `http`/`https`. | | `robots.txt denied` | `robots_txt: true` and the page disallows fetching. | | `upstream service unavailable` | The upstream answered with an unexpected status code. | | `every url failed to be scraped; the pages may be unreachable or hold no extractable text` | All URLs failed. Individual causes are logged to `stderr`, not returned. | | `every query failed; the search upstream may be unreachable` | All queries failed. | | `internal server error` | Anything unclassified. | The all-failed messages deliberately do not distinguish timeouts from other causes: a mixed batch can fail for several reasons at once, and the per-item `status` already carries that detail whenever at least one item survives. ### Known limitations - **`web_scrape` handles HTML only.** Pages are run through a readability extractor, which needs article markup, so `text/plain` responses yield nothing and come back as `status: "failed"`. Raw-file hosts are the common case: `raw.githubusercontent.com`, `github.com/.../raw/...`, `cdn.jsdelivr.net`. Scrape the rendered page instead of the raw file. - **`web_search_images` relevance is not guaranteed.** For some queries Bing Images serves a page that is not a result set, and it is parsed as though it were — the tool then returns unrelated images with `status: "success"`. Treat image results as best-effort and verify them before showing them to a user. - **No JavaScript.** Pages are fetched as-is; content rendered client-side is invisible to the extractor. --- ## Quick start ### Install Pick whichever fits — all of them give the identical server. **Container** (no Go toolchain needed): ```bash docker pull ghcr.io/role1776/mcp-retrieval:latest ``` **Prebuilt binary** — grab the archive for your platform from the [latest release](https://github.com/Role1776/mcp-retrieval/releases/latest), unpack it, and put `mcp-retrieval` on your `PATH`. **MCP Bundle** — for clients that install `.mcpb` files, download `mcp-retrieval_<version>_<os>_<arch>.mcpb` from the [latest release](https://github.com/Role1776/mcp-retrieval/releases/latest) and open it with your client. The bundle carries the compiled binary, so it needs neither Docker nor Go. Pick the file matching your OS *and* CPU architecture: a bundle holds one native binary. **From source:** ```bash go install github.com/Role1776/mcp-retrieval/app/cmd/mcp-retrieval@latest # needs Go 1.25.5+ ``` Or build the binary in place (the Go module lives in `app/`): ```bash make build # -> bin/mcp-retrieval ``` ### Run ```bash # defaults: stdio transport, no configuration needed ./bin/mcp-retrieval # with an explicit env file ./bin/mcp-retrieval -env /absolute/path/to/.env ``` The one flag is optional: | Flag | Meaning | | :--- | :--- | | `-env` | Path to a `.env` file. If omitted — or if the file does not exist — the server starts on defaults and whatever is already in the environment. There is no implicit lookup: under stdio the working directory is chosen by the MCP client, so a relative default would be unpredictable. | ### Connecting an MCP client (stdio) Point your client at the built binary. Example Claude Desktop config: ```json { "mcpServers": { "retrieval": { "command": "/absolute/path/to/mcp-retrieval", "env": { "MAX_RESULTS": "20" } } } } ``` The `env` block is optional — `"command"` alone is enough. ### Connecting an MCP client (container) Run the image on stdio. Configuration still travels through the `env` block, but Docker needs each variable named on the command line with `-e` for it to reach the process: ```json { "mcpServers": { "retrieval": { "command": "docker", "args": [ "run", "-i", "--rm", "-e", "MAX_RESULTS", "-e", "DEFAULT_TIMEOUT_MS", "ghcr.io/role1776/mcp-retrieval:latest" ], "env": { "MAX_RESULTS": "20", "DEFAULT_TIMEOUT_MS": "5000" } } } } ``` `-i` is required — without it the container gets no stdin and the client sees the server die immediately. Clients that install from the [MCP Registry](https://registry.modelcontextprotocol.io) build this invocation themselves and prompt for the variables declared in [`server.json`](server.json). ### Running over HTTP Set `MCP_TRANSPORT=http` and the server listens on `SERVER_PORT` at `MCP_PATH` (default `http://localhost:8080/mcp`). --- ## Configuration Everything is configured through **environment variables**, and each value is validated before startup: a non-numeric or non-positive value is a startup error. Relationships *between* limits are not checked at startup — see [Limits](#limits). Variables already present in the environment win over a `.env` file, so an MCP client's `env` block always takes effect. Every field has a sensible default, so the server runs with no configuration at all (stdio transport). See [`.env.example`](.env.example) for the full list at its default values, ready to copy to `.env`. ### MCP server | Env | Default | Notes | | :--- | :--- | :--- | | `MCP_TRANSPORT` | `stdio` | `stdio` or `http`. | | `MCP_NAME` | `mcp-retrieval` | Server name advertised to clients. | | `MCP_PATH` | `/mcp` | HTTP route (http transport only). | The version advertised to clients is not configurable: it is stamped into the binary at build time from the git tag. ### HTTP server (http transport only) | Env | Default | | :--- | :--- | | `SERVER_PORT` | `8080` | | `SERVER_READ_TIMEOUT` | `60s` | | `SERVER_WRITE_TIMEOUT` | `60s` | ### HTTP client and proxy | Env | Default | Notes | | :--- | :--- | :--- | | `MAX_IDLE_CONNS_PER_HOST` | `100` | HTTP connection pooling. | | `PROXY_HOST` | — | Optional. If set, requests are routed through a rotating-session proxy. | | `PROXY_PORT` | — | Required when `PROXY_HOST` is set. | | `PROXY_SCHEME` | — | Required when `PROXY_HOST` is set. | | `PROXY_LOGIN` | — | Required when `PROXY_HOST` is set. | | `PROXY_PASSWORD` | — | Required when `PROXY_HOST` is set. | When a proxy is configured, each outbound request gets a unique session id appended to the login, so the upstream provider rotates the exit IP per request. ### Limits | Env | Default | | :--- | :--- | | `MAX_QUERIES` | `10` | | `DEFAULT_RESULTS` | `5` | | `MAX_RESULTS` | `20` | | `DEFAULT_TIMEOUT_MS` | `5000` | | `MAX_TIMEOUT_MS` | `10000` | | `MIN_TIMEOUT_MS` | `1000` | | `DEFAULT_IMAGES` | `5` | | `MAX_IMAGES` | `10` | | `DEFAULT_DOCUMENT_CHARS` | `20000` | | `MAX_DOCUMENT_CHARS` | `20000` | Each value is checked on its own — it must be greater than zero — but the `DEFAULT_*`, `MIN_*` and `MAX_*` triples are **not** cross-checked against each other at startup. An inconsistent set does not stop the server; it is reconciled per request instead: - a value the caller omits, or passes as zero or negative, falls back to the matching `DEFAULT_*`; - the result is then clamped into `[MIN_*, MAX_*]`, so a `DEFAULT_*` larger than its `MAX_*` simply yields `MAX_*`; - if `MIN_*` exceeds `MAX_*`, the maximum wins. The effective limit is therefore always within the configured maximum, and misconfiguration degrades to a working server rather than a failed start. The trade-off is that it degrades **silently**: a typo such as `MAX_RESULTS=2` instead of `20` produces no warning, only quietly smaller responses. Worth double-checking these values when results look truncated. ### Logging | Env | Default | Notes | | :--- | :--- | :--- | | `LOG_MODE` | `local` | `local` → text handler at debug level; `prod` → JSON handler at info level. Logs go to `stderr`. | --- ## Architecture The project follows a clean, layered structure. Dependencies point inward toward the domain, and each layer talks to the next through interfaces. ``` app/ the Go module: sources plus its build files (Dockerfile, .dockerignore, .goreleaser.yaml) cmd/mcp-retrieval/main.go entry point: parse flags, load config, run app internal/ app/ wiring + lifecycle (build server, run, graceful shutdown) config/ config loading (.env → env vars → validate) domain/ core types (Query, Link, Document, Snippet, Image) and errors dto/web/ request/response shapes for the MCP tools transport/mcp/ MCP layer router/ registers every tool group on the MCP server web/ tool handlers utils/ schema helpers and error → tool-result mapping usecase/web/ business logic: validation, parallelism, timeouts, dedupe/limit/rerank adapter/web/ retrieval-go client wiring (search, images, scrape, proxy) pkg/ reusable building blocks (mcpserver, server, logger, validator) ``` Request flow for a tool call: ``` MCP client → transport/mcp/web (handler) → usecase/web → adapter/web → retrieval-go → the web ↑ maps errors ↑ validates, fans out, limits results ``` Search and scrape both fan out across the input list concurrently and aggregate per-item results, each with its own status (`success`, `failed`, `timeout`). A call only fails outright when **every** item in it fails. --- ## Retrieval engine All network work is delegated to [`retrieval-go`](https://github.com/free-llms-foundation/retrieval-go), configured in [`app/internal/adapter/web`](app/internal/adapter/web/retrieval.go). Worth knowing: - **Sources.** Web search uses **DuckDuckGo Lite**; image search uses **Bing Images**; page fetching runs the raw HTML through a **readability** extractor and converts the main article to **Markdown** (tables included). No search-engine API keys are required. - **Browser impersonation.** The adapter enables `WithBrowserRotation()`, so each request is sent from one of ~11 real browser profiles picked at random. Every profile pairs a genuine **TLS/JA3 fingerprint** (via [uTLS](https://github.com/refraction-networking/utls)) with a matching `User-Agent` and client-hint headers — Chrome 133/131/120 (Windows/macOS/Linux), Edge 131, Firefox 120 (Windows/macOS), Safari 18.4 (macOS), and iOS 18.4 Safari. This makes the traffic look like ordinary browsers rather than a Go HTTP client, which is what keeps the free sources reachable. - **Proxy rotation.** When `PROXY_HOST` is configured, the adapter installs a proxy factory that appends a unique `session-<id>` to the proxy username on every request. With a session-based residential/rotating proxy provider, that yields a **fresh exit IP per request**, spreading load and avoiding rate limits. Without a proxy, requests go out directly. - **Response handling.** Responses are transparently decompressed (`gzip`, `br`, `zstd`, `deflate`), and keep-alive is disabled (`WithDisableKeepAlive()`) so pooled connections don't pin a single fingerprint/IP across requests. None of this needs configuration to work — the defaults above are applied automatically. Only proxy credentials are optional extras. ## Development Everything Go lives in `app/`, so either use the makefile from the repository root or pass `-C app` to the toolchain: ```bash make build # compile the binary make test # run tests go -C app build ./... # compile everything go -C app test ./... # run tests go -C app vet ./... # static checks ``` See [`CONTRIBUTING.md`](CONTRIBUTING.md) for pull-request guidelines. ## License Released under the [MIT License](LICENSE).