Content
# websearch-mcp
Local-first MCP server for web search and full-page reading. It is designed to sit beside a local LLM client such as a llama.cpp WebUI and do only the network and document work:
- search via pluggable providers
- fetch pages over plain HTTP first
- fall back to a browser for JS-heavy pages
- extract text from PDF documents
- convert cleaned page content into markdown
- chunk long pages so the model can read them incrementally
The server keeps the normal tool surface small and local-first. `search_web` and `read_url` only do network and document work. `deep_research` is the exception: it can call a local OpenAI-compatible endpoint such as `llama.cpp` internally to plan and condense evidence.
## Current tools
- `search_web`
- `read_url`
- `deep_research`
- `deep_research_status`
## Current providers
- `duckduckgo`
- `searxng`
The provider interface is intentionally small so additional adapters such as Brave, Serper, Tavily, Exa, or Google CSE can be added without changing the MCP layer.
## Quick start
```powershell
npm install
Copy-Item .env.example .env
npm run dev
```
Requirements:
- Node.js `22+`
- If you want browser fallback for JS-heavy pages, install a Playwright browser too:
```powershell
npx playwright install chromium
```
Default bind:
- host: `127.0.0.1`
- port: `8642`
- MCP endpoint: `http://127.0.0.1:8642/mcp`
Quick checks:
```powershell
curl http://127.0.0.1:8642/healthz
npm run smoke:search
```
The smoke search uses the current default provider from `.env`.
## Using with a WebUI
Typical local setup:
1. Start your `llama.cpp` server or other local model backend.
2. Start this MCP server with `npm run dev` or `npm start`.
3. In your UI, add `http://127.0.0.1:8642/mcp` as a Streamable HTTP MCP server.
For browser-based UIs, set `CORS_ALLOW_ORIGIN` if you want to restrict which local origin can connect.
## Environment
`DEFAULT_SEARCH_PROVIDER`
- Default: `duckduckgo`
- Supported now: `duckduckgo`, `searxng`
`SEARXNG_URL`
- Required only when `searxng` is used.
- Example: `https://search.example.com`
`CORS_ALLOW_ORIGIN`
- Default: `*`
- Needed for browser-based MCP clients such as WebUIs running on another local port.
- Set this to a specific origin such as `http://127.0.0.1:8080` if you want to tighten it down.
`BROWSER_FALLBACK`
- Default: `true`
- When enabled, `read_url` retries weak extractions with Playwright.
`SEARCH_MIN_INTERVAL_MS`
- Default: `2500`
- Adds baseline search pacing so agent loops do not hammer DuckDuckGo immediately.
`SEARCH_RETRY_ATTEMPTS`
- Default: `3`
- How many times DuckDuckGo search will retry after temporary blocking or rate-limit style responses.
`SEARCH_RETRY_BACKOFF_MS`
- Default: `4000`
- Base backoff delay used before retrying blocked DuckDuckGo searches.
`SEARCH_RETRY_BACKOFF_MAX_MS`
- Default: `15000`
- Upper bound for the adaptive backoff delay.
`SEARCH_BLOCK_COOLDOWN_MS`
- Default: `90000`
- After DuckDuckGo returns an anti-bot challenge repeatedly, the provider enters a cooldown window and rejects new DuckDuckGo requests quickly instead of re-hammering the endpoint.
`LLM_BASE_URL`
- Default: `http://localhost:8080`
- Used by the `deep_research` tool for internal planning and evidence condensation.
- Not required if you only use `search_web` and `read_url`.
`LLM_MODEL`
- Optional override.
- If unset, `deep_research` tries to auto-detect the currently loaded model from `GET /v1/models`.
`LLM_API_KEY`
- Optional bearer token for OpenAI-compatible endpoints that require auth.
`LLM_TIMEOUT_MS`
- Default: `120000`
- Timeout for internal LLM calls made by `deep_research`.
`DEEP_RESEARCH_MAX_QUERIES`
- Default: `5`
- Default upper bound for internally generated search queries.
`DEEP_RESEARCH_MAX_PAGES`
- Default: `6`
- Default upper bound for pages opened during one research run.
`DEEP_RESEARCH_MAX_FOLLOWUP_LINKS`
- Default: `3`
- Default upper bound for reference/follow-up pages opened after the initial search results.
`DEEP_RESEARCH_SEARCH_RESULTS_PER_QUERY`
- Default: `5`
- Default search result count considered for each generated query.
`DEEP_RESEARCH_MAX_DURATION_MS`
- Default: `45000`
- Hard time budget for one deep research run. The tool returns partial evidence rather than continuing indefinitely.
`PLAYWRIGHT_BROWSER`
- Default: `chromium`
- Supported by this scaffold: `chromium`, `firefox`, `webkit`
`PLAYWRIGHT_EXECUTABLE_PATH`
- Optional path to an installed browser binary.
## Research workflow
For deeper research, do not stop at one search result:
1. Run several `search_web` queries from different angles.
2. Use `read_url` on multiple domains.
3. Follow the `followUpLinks` returned by `read_url` to inspect references, source documents, and cited studies.
4. Use `chunk_index` in `read_url` if you need later sections of a long page.
5. Only then write the report.
If your client model is weak at tool orchestration, use `deep_research` instead. That tool performs the search/read/follow-up loop inside the MCP server and returns an evidence bundle with concise findings rather than a fully polished final report.
`deep_research` now runs in the background:
1. Call `deep_research` to start a job.
2. It returns a `jobId` immediately, plus an estimated runtime and a recommended wait before polling.
3. Tell the user the job is running and wait roughly that long before calling `deep_research_status`.
4. When complete, the evidence bundle is returned in the status response.
## Notes
- The server is local-only by default and validates `Host` against `HOST_ALLOWLIST`.
- The server sends CORS headers for browser-based MCP clients and exposes the MCP session/protocol headers.
- DuckDuckGo search uses the HTML results endpoint with an explicit browser-style user agent.
- DuckDuckGo retries temporary anomaly/rate-limit style failures with adaptive backoff before surfacing an error.
- When DuckDuckGo presents a bot challenge, `search_web` now returns a structured blocked response with `retryAfterMs` instead of a raw tool failure, and deep research stops issuing more DuckDuckGo queries during the cooldown window.
- If you copied `.env` before recent changes, update `USER_AGENT` and the `SEARCH_*` settings.
- `read_url` returns metadata, one chunk at a time, and ranked follow-up links. Use `chunk_index` to request later chunks from the same page.
- `read_url` can now extract text from PDFs served as `application/pdf` or `.pdf` URLs.
- `deep_research` tries to auto-detect the loaded llama.cpp model from `/v1/models` when `LLM_MODEL` is not set.
- `deep_research` establishes the current time internally and does not require a separate date tool.
- `deep_research` returns structured evidence, concise findings, and source coverage instead of a final polished report.
- `deep_research` filters obvious search-engine pages and DOI redirect pages so they do not count as evidence sources.
- Background deep research jobs are currently in-memory only, so they are lost if the MCP server restarts.
- Cache files are written into `.cache/`.
## Architecture
`src/index.ts`
- Express entrypoint
- Streamable HTTP MCP endpoint
- per-request MCP server creation
`src/mcp/createServer.ts`
- tool registration
`src/providers/`
- search provider interface
- `duckduckgo`
- `searxng`
`src/services/`
- search orchestration
- page fetching and conversion
- local LLM client for deep research
- background deep research job manager
- file cache
## Next additions
- authenticated commercial providers
- per-domain throttling and robots controls
- deduplication across providers
- richer result metadata
## Sources used for the scaffold
- MCP TypeScript SDK repo and docs: https://github.com/modelcontextprotocol/typescript-sdk
- Current v1 release shown in the official repo release list: https://github.com/modelcontextprotocol/typescript-sdk
