Content
[中文](README-zh.md)
-----
# mcp-server-requests
An MCP server that provides HTTP request capabilities, enabling LLMs to fetch and process web content.
## Features
- **fetch**: Web content fetching tool
- Supports converting web content to Markdown format
- Supports filtering web content (non-visual elements like script, style, meta)
- Feeds into LLM context, reducing token consumption through Markdown conversion or filtering non-visual elements
- **fetch_to_file**: Web content saving tool
- Web content is not fed into LLM context, avoiding excessive token consumption
- **http_request**: Generic HTTP request tool
- Full HTTP method support (GET, POST, PUT, PATCH, DELETE)
- Supports custom request headers
- Supports text and JSON request bodies
- Returns complete HTTP response information (status, headers, body) for LLMs
- Supports custom User-Agent
- Supports random User-Agent generation
- Supports using MCP Root capability, controlled by `--use-root`, usable with MCP Clients that have `root` capability
## Installation
```bash
git clone https://github.com/coucya/mcp-server-requests.git
cd mcp-server-requests
pip install .
```
## Usage
### MCP Server Configuration
```json
{
"mcpServers": {
"mcp-server-requests": {
"command": "python",
"args": [
"-m",
"mcp_server_requests"
]
}
}
}
```
### Command Line
### 0. **Start MCP Server**
Start the MCP server directly:
```bash
python -m mcp_server_requests
```
#### Options
- `--user-agent TEXT`: Specify custom User-Agent string
- `--random-user-agent [browser=xxx;os=xxx]`: Use randomly generated User-Agent
- `--force-user-agent`: Force using command line specified User-Agent, ignoring LLM provided UA
- `--list-os-and-browser`: List available browsers and OS for random User-Agent generation
- `--use-root`: Enable workspace root support for file operations (requires MCP Client with `root` capability support)
- `--allow-external-file-access`: Allow file operations outside workspace (only with --use-root)
#### Option Details
- `--user-agent` and `--random-user-agent` are mutually exclusive and cannot be used together
- User-Agent setup methods:
- Custom string: `--user-agent "Mozilla/5.0 (...)"`
- Fully random: `--random-user-agent`
- Conditional random generation:
- Specify browser type: `--random-user-agent browser=chrome`
- Specify OS: `--random-user-agent os=windows`
- Both browser and OS: `--random-user-agent browser=chrome;os=windows`
- Note: Browser and OS parameters are case insensitive
- Use `--list-os-and-browser` to view available browsers and OS for `--random-user-agent`.
- `--force-user-agent` option controls User-Agent priority:
- When `--force-user-agent` is enabled: Force using command-line specified User-Agent (via `--user-agent` or `--random-user-agent`)
- When `--force-user-agent` is disabled:
- If LLM provides User-Agent in `headers`, use the LLM-provided one
- Otherwise use command-line specified User-Agent
- If none specified, use default User-Agent: `Mozilla/5.0 (compatible; mcp-server-requests/{version})`
- `--use-root` enables file operations relative to workspace root:
- **Prerequisite**: Requires MCP Client with `root` capability support
- When enabled: `fetch_to_file` allows absolute paths or relative paths, relative paths are relative to workspace root
- When disabled: `fetch_to_file` must use absolute paths, files are saved to specified absolute path locations
- `--allow-external-file-access`: Controls whether file operations outside workspace are allowed (only usable with --use-root)
- **Prerequisite**: Requires `--use-root` to be enabled, this option has no effect if `--use-root` is not enabled
- When enabled: `fetch_to_file` can operate on files outside the workspace directory
- When disabled: `fetch_to_file` can only operate on files within the workspace directory, even if absolute paths are provided, files outside the workspace cannot be operated on
---
### 1. **fetch - Fetch Web Content**
The fetch subcommand is equivalent to the MCP `fetch` tool functionality, used to demonstrate fetch capabilities.
```bash
python -m mcp_server_requests fetch <URL> [--return-content {raw,basic_clean,strict_clean,markdown}]
```
**Options:**
- `--return-content`: Return content type (default: markdown)
- **raw**: Return raw unprocessed HTML content (includes response headers)
- **basic_clean**: Basic cleanup, removing non-display tags (script, style, meta, etc.) while preserving structure
- **strict_clean**: Strict cleanup, removing non-display tags and most HTML attributes, keeping only essential structure
- **markdown**: Convert HTML to clean Markdown format
---
## Functionality
### MCP Tool Details
#### 1. **fetch** - Fetch Web Content
Fetch content from the specified URL and return it in the specified format.
**Parameters:**
- **url** (required, string): Target URL, supports HTTP/HTTPS protocols
- **return_content** (optional, enum): Content processing format, defaults to "markdown"
- `"raw"`: Return raw HTML content with complete HTTP response headers
- `"basic_clean"`: Basic cleanup, removing non-display tags (script, style, meta, etc.) while preserving HTML structure
- `"strict_clean"`: Strict cleanup, removing non-display tags and most HTML attributes, keeping only essential structure
- `"markdown"`: Convert HTML content to clean Markdown format
**Returns:** Content processed according to the specified format
---
#### 2. **fetch_to_file** - Fetch Web Content and Save to File
Fetch content from the specified URL and save it to a file.
**Parameters:**
- **url** (required, string): Target URL
- **file_path** (required, string): File save path
- When `--use-root` is enabled: Must be a **relative path** (relative to workspace root)
- When `--use-root` is disabled: Must be an **absolute path**
- **return_content** (optional, enum): Content processing format, defaults to "markdown", same options as fetch tool
**Features:**
- Automatically creates parent directories for nested paths
- All files saved with UTF-8 encoding
- When `--use-root` is enabled, can work with `--allow-external-file-access` to control file access scope
**Returns:** Operation result or error message
---
#### 3. **http_request** - Generic HTTP Request Tool
Send HTTP requests using any method and get complete responses.
**Parameters:**
- **url** (required, string): Target URL
- **method** (optional, enum): HTTP method, defaults to "GET"
- `"GET"`, `"POST"`, `"PUT"`, `"PATCH"`, `"DELETE"`
- **query** (optional, object): URL query parameters as key-value pairs, automatically URL-encoded
- **headers** (optional, object): Custom HTTP request headers
- **data** (optional, string): Text format request body data, mutually exclusive with json parameter
- **json** (optional, any type): JSON format request body data, mutually exclusive with data parameter
**Parameter Constraints:**
- `data` and `json` parameters cannot be used together
- When using `json` parameter, `Content-Type: application/json` is automatically set
- When using `data` parameter, it's recommended to manually set appropriate `Content-Type`
**Returns:** Complete HTTP response (status line, response headers, and response body)
---
**Note**: Individual HTTP method tools (http_get, http_post, http_put, http_patch, http_delete) have been removed and replaced by the unified **http_request** tool.
## License
MIT
