Content
# Crawl4AI MCP Server
A web crawler MCP server using crawl4ai that allows AI models to crawl websites and extract content in markdown format.
## Features
- **Recursive Crawling**: Recursively crawl websites starting from a URL
- **Configurable Depth**: Control how deep the crawler should go
- **Page Limit**: Set maximum number of pages to crawl
- **Timeout Configuration**: Set custom timeout for long crawling operations
## Installation
### Using uv (Recommended)
The fastest way to install and use the Crawl4AI MCP server is with uv:
```bash
# Install directly from GitHub
uv pip install git+https://github.com/Kavin-kumar10/crawl4ai-mcp.git
# Or create a virtual environment first
uv venv -p python3.10 .venv
source .venv/bin/activate # On Unix/macOS
# or
.venv\Scripts\activate # On Windows
uv pip install git+https://github.com/Kavin-kumar10/crawl4ai-mcp.git
```
### Using pipx
You can also install it with pipx for an isolated environment:
```bash
pipx install git+https://github.com/Kavin-kumar10/crawl4ai-mcp.git
```
This will install the package in an isolated environment and make the `crawl4ai-mcp` command available globally.
### Using pip
You can also install it with pip:
```bash
pip install git+https://github.com/Kavin-kumar10/crawl4ai-mcp.git
```
### From Source
To install from source:
```bash
git clone https://github.com/Kavin-kumar10/crawl4ai-mcp.git
cd crawl4ai-mcp
# Using uv (recommended)
uv pip install -e .
# Or using pip
pip install -e .
```
## Usage with MCP
### Running the Server
Once installed, you can run the MCP server directly:
```bash
# If installed with pipx
pipx run crawl4ai-mcp
# If installed with pip or uv
python -m Crawl_mcp.main
```
The server uses stdio transport (not URL-based) for communication with MCP clients.
### Using the mcp.json Configuration
This repository includes an `mcp.json` file that you can use to configure the MCP server in MCP-compatible clients:
```bash
# Copy it to your home directory or project directory
cp mcp.json ~/.mcp.json
```
The `mcp.json` file contains:
- Server configuration with stdio transport
- Tool definitions with parameters and return types
- Descriptions for tools and parameters
### Available Tools
The MCP server provides the following tool:
#### crawl_recursive
Recursively crawl a website starting from a URL.
**Parameters:**
- `url` (string): The URL to start crawling from
- `max_depth` (integer, optional): Maximum depth for recursive crawling (default: 2)
- `max_pages` (integer, optional): Maximum number of pages to crawl (default: 500)
**Example:**
```json
{
"url": "https://example.com",
"max_depth": 3,
"max_pages": 100
}
```
### Example Usage with Claude
Here's an example of how to use the MCP server with Claude after configuring it in your MCP environment:
```
I need to use the Crawl4AI MCP server to crawl a website.
<use_mcp_tool>
<server_name>crawl4ai-mcp</server_name>
<tool_name>crawl_recursive</tool_name>
<arguments>
{
"url": "https://example.com",
"max_depth": 2,
"max_pages": 100
}
</arguments>
</use_mcp_tool>
```
Note: The MCP server uses stdio transport for communication, not HTTP/URL-based transport. This means it runs in the terminal and communicates through standard input/output streams.
## Development
### Requirements
- Python 3.10 or higher
- MCP CLI 1.7.1 or higher (`pip install mcp[cli]>=1.7.1`)
- crawl4ai (`pip install crawl4ai`)
### Development Setup
Set up your development environment:
```bash
# Clone the repository
git clone https://github.com/Kavin-kumar10/crawl4ai-mcp.git
cd crawl4ai-mcp
# Create a virtual environment with uv
uv venv -p python3.10 .venv
source .venv/bin/activate # On Unix/macOS
# or
.venv\Scripts\activate # On Windows
# Install in development mode
uv pip install -e .
# Install development dependencies (if you have any)
uv pip install -e ".[dev]"
```
### Testing
To test the MCP server locally:
```bash
# Run the server in development mode with the MCP Inspector
mcp dev Crawl_mcp/server.py
# Or run the module directly (stdio mode)
python -m Crawl_mcp.main
```
When running in stdio mode, the server expects JSON-RPC messages on stdin and writes responses to stdout. This is how MCP clients communicate with the server.
### Dependency Management
Use uv to manage dependencies:
```bash
# Check for dependency conflicts
uv pip check
# Generate a lock file (if you have a requirements.in file)
uv pip compile requirements.in -o requirements.txt
```
## License
[MIT License](LICENSE)
You Might Also Like
MarkItDown MCP
markitdown-mcp is a lightweight MCP server for converting various URIs to Markdown.
Github
GitHub MCP Server connects AI tools to GitHub for code management and automation.

apisix
Apache APISIX is an API Gateway for managing APIs and microservices.
opik
Opik is a powerful tool for managing and optimizing machine learning experiments.

MCP Toolbox for Databases
MCP Toolbox for Databases is an open-source server simplifying database tool...

sqlglot
SQLGlot is a no-dependency SQL parser and transpiler supporting 30 dialects.