Content
# Scrapling MCP Server
> **Scrapling MCP** — A universal Web scraping MCP server based on [Scrapling](https://github.com/D4Vinci/Scrapling). Three-engine architecture covering all scenarios from static pages to Cloudflare-protected sites, supporting JS rendering, anti-scraping bypass, cookie injection, and 2FA TOTP automation.
[](https://github.com/D4Vinci/Scrapling)
[](https://modelcontextprotocol.io)
[](LICENSE)
[](https://www.python.org/downloads/)
[](https://docs.astral.sh/uv/)
---
## 🔥 Hot Recommendations
<!-- 20 recommended titles -->
| # | Title | Keywords |
|---|---|---|
| 01 | **The strongest Web scraping tool in the AI era? Scrapling MCP full review** | AI, Scraping, MCP |
| 02 | **Claude Code directly scrapes any webpage! MCP protocol practical** | Claude Code, MCP |
| 03 | **Goodbye blocked! Scrapling three-engine bypass Cloudflare Turnstile** | Cloudflare, anti-scraping |
| 04 | **How do ChatGPT/Claude scrape webpage content in real-time? MCP server details** | ChatGPT, Claude, MCP |
| 05 | **The most worth-attending Python open-source crawler framework in 2026: Scrapling** | Python, crawler, open-source |
| 06 | **Building AI crawlers from scratch: Scrapling MCP Server complete tutorial** | tutorial, crawler, AI |
| 07 | **Still using BeautifulSoup? Scrapling is 774 times faster** | BeautifulSoup, performance |
| 08 | **The ultimate solution to bypass Cloudflare verification (no paid API)** | Cloudflare, free |
| 09 | **x.com (Twitter) data scraping practical: Cookie injection + browser rendering** | x.com, data scraping |
| 10 | **OpenCode/Codex CLI/Claude Code three-end MCP configuration guide** | OpenCode, Codex, Claude |
| 11 | **Selenium is dead? Scrapling stealth browser mode comparison** | Selenium, comparison |
| 12 | **SPA page scraping solution for Yuque/Notion/Zhihu** | Yuque, SPA, JS rendering |
| 13 | **reCAPTCHA v3 Enterprise bypass practical: StealthyFetcher configuration** | reCAPTCHA, anti-detection |
| 14 | **Web Scraping avoidance guide: 2026 anti-scraping technology panorama** | anti-scraping, technology analysis |
| 15 | **2FA TOTP automatic verification: pyotp + Scrapling login automation** | 2FA, TOTP, automation |
| 16 | **How to automate scraping of end-to-end encrypted web pages (WhatsApp/Signal)** | E2EE, WhatsApp, Signal |
| 17 | **uvx vs pip: Python package management new era choice** | uv, pip, Python |
| 18 | **AI Agent must-have: MCP protocol to achieve Web scraping capability** | AI Agent, MCP |
| 19 | **The Swiss Army knife for crawler engineers: Scrapling MCP full-scene test report** | crawler, test |
| 20 | **From HTTP to stealth browser: Scrapling three-layer engine design philosophy** | architecture, design |
> 💡 All of the above titles can be directly published on platforms like Zhihu, Jujun, V2EX, Open China, CSDN, etc.
---
## 📋 Overview
Scrapling MCP Server exposes Scrapling's Web scraping capabilities to AI clients (such as Claude Code, OpenCode, Codex CLI, etc.) through the MCP (Model Context Protocol) standard interface, allowing AI to directly scrape and extract any webpage content.
### Core Features
| Feature | Description |
|---|---|
| **Three-engine architecture** | `get` (static) → `fetch` (browser) → `stealthy_fetch` (stealth) |
| **Cloudflare bypass** | Automatically solves Turnstile/Interstitial verification |
| **JS dynamic rendering** | Based on Playwright Chromium, complete JavaScript execution |
| **CSS precise extraction** | Supports CSS selectors, returns only target elements |
| **Batch parallel** | `bulk_get` / `bulk_fetch` / `bulk_stealthy_fetch` multi-URL concurrency |
| **Anti-fingerprint detection** | TLS fingerprint disguise, Canvas noise, WebRTC shielding |
| **Cookie injection** | Supports extracting logged-in cookies from Chrome |
| **Multi-format output** | HTML/Markdown/plain text three output formats |
| **Proxy support** | HTTP proxy + DNS-over-HTTPS anti-leak |
| **Ad blocking** | Built-in 3500+ ad domain blocking |
---
## 🔧 Quick Start
### Prerequisites
- Python 3.10+
- [uv](https://docs.astral.sh/uv/) (recommended) or pip
- Optional: Google Chrome (for `--real-chrome` mode)
### Installation
```bash
# 1. Install Scrapling (including MCP and browser dependencies)
pip install "scrapling[ai]" # or uv pip install "scrapling[ai]"
# 2. Download browser dependencies (first time)
scrapling install --force
```
### MCP Configuration
Add the following to the MCP configuration file of the AI client:
```json
{
"mcpServers": {
"scrapling": {
"type": "stdio",
"command": "uvx",
"args": [
"--from",
"scrapling[ai]",
"scrapling",
"mcp"
]
}
}
}
```
> **Note:** Must use `scrapling[ai]` (with extras), otherwise, the `click` and `mcp` dependencies are missing, leading to connection failure.
### Client Configuration Paths
| Client | Configuration File Path |
|---|---|
| **Claude Desktop** | `~/Library/Application Support/Claude/claude_desktop_config.json` |
| **Claude Code** | `.claude/settings.local.json` + project `.mcp.json` |
| **OpenCode** | `opencode mcp add scrapling -- uvx --from "scrapling[ai]" scrapling mcp` |
| **Codex CLI** | `codex mcp add scrapling -- uvx --from "scrapling[ai]" scrapling mcp` |
| **Cursor** | project `.cursor/mcp.json` |
| **Windsurf** | project `.windsurf/mcp.json` |
---
## 🚀 Usage Guide
### Through AI Conversation (MCP Protocol)
Directly describe the requirements in the AI conversation, and the AI will automatically choose the appropriate tool:
```
Help me scrape the content of https://example.com and convert it to Markdown
→ AI calls get()
Bypass Cloudflare to scrape https://blocked-site.com
→ AI calls stealthy_fetch(solve_cloudflare: true)
Scrape this SPA page after browser rendering
→ AI calls fetch()
```
### Through CLI
```bash
# Static page
scrapling extract get --ai-targeted "https://example.com" output.md
# Browser rendering
scrapling extract fetch --headless --network-idle "https://spa-site.com" output.md
# Cloudflare bypass
scrapling extract stealthy-fetch --solve-cloudflare "https://cf-site.com" output.md
# CSS precise extraction
scrapling extract get -s "article.content" "https://example.com" output.html
# POST request
scrapling extract post -j '{"key":"value"}' "https://api.example.com" output.md
```
### Through Python API
```python
from scrapling.fetchers import DynamicFetcher
result = DynamicFetcher.fetch(
"https://example.com",
headless=True,
network_idle=True,
main_content_only=True,
timeout=60000,
)
print(result.text)
```
---
## 📚 Tool Details
### Layer 1: `get` — Static HTTP
Based on `curl_cffi`, simulate browser TLS fingerprint + HTTP headers.
| Parameter | Type | Default | Description |
|---|---|---|---|
| `url` | string | — | Target URL (required) |
| `css_selector` | string | — | CSS selector |
| `extraction_type` | string | `markdown` | `markdown` / `html` / `text` |
| `main_content_only` | bool | `false` | Extract main content only |
| `timeout` | int | `30` | Timeout in seconds |
| `impersonate` | string | `chrome` | Browser fingerprint |
| `proxy` | string | — | HTTP proxy |
| `headers` | object | — | Custom request headers |
| `cookies` | string | — | Cookie string |
### Layer 2: `fetch` — Dynamic Browser
Based on Playwright Chromium, complete JavaScript execution.
| Parameter | Type | Default | Description |
|---|---|---|---|
| `url` | string | — | Target URL |
| `wait_selector` | string | — | Wait for element to appear |
| `network_idle` | bool | `false` | Wait for network idle |
| `block_ads` | bool | `false` | Block ads |
| `disable_resources` | bool | `false` | Disable images and fonts |
| `real_chrome` | bool | `false` | Use system Chrome |
| `locale` | string | — | Browser language |
### Layer 3: `stealthy_fetch` — Stealth Browser
Based on StealthyFetcher, increase anti-detection capabilities.
| Parameter | Type | Default | Description |
|---|---|---|---|
| `solve_cloudflare` | bool | `false` | ⭐ Solve Cloudflare verification |
| `block_webrtc` | bool | `false` | Block WebRTC |
| `hide_canvas` | bool | `false` | Canvas fingerprint noise |
| `allow_webgl` | bool | `true` | WebGL control |
### Bulk Series
| Tool | Description |
|---|---|
| `bulk_get` | Parallel HTTP scraping of multiple URLs |
| `bulk_fetch` | Parallel browser rendering of multiple URLs |
| `bulk_stealthy_fetch` | Parallel stealth scraping of multiple URLs |
---
## 🔥 Advanced Scenarios
### x.com (Twitter) Scraping
```python
import browser_cookie3
from scrapling.fetchers import DynamicFetcher
# Extract logged-in x.com cookie from Chrome
cj = browser_cookie3.chrome(domain_name="x.com")
cookies = [{"name": c.name, "value": c.value, "domain": ".x.com", "path": "/"} for c in cj]
result = DynamicFetcher.fetch(
"https://x.com/elonmusk",
headless=True,
network_idle=True,
timeout=180000,
cookies=cookies,
wait=5000,
)
```
### reCAPTCHA v3 Enterprise Bypass
```json
stealthy_fetch(
url: "https://target.com",
solve_cloudflare: true,
block_webrtc: true,
hide_canvas: true,
allow_webgl: false,
headless: false
)
```
### 2FA TOTP Automatic Generation
```python
import pyotp, base64, binascii
# hex format key
secret_hex = "your 32-bit hex key"
raw = binascii.unhexlify(secret_hex)
secret_b32 = base64.b32encode(raw).decode()
totp = pyotp.TOTP(secret_b32)
code = totp.now() # → 6-digit verification code
```
### E2EE Application Session Persistence
```json
open_session(name: "whatsapp", headless: false)
// First scan → subsequent automation
```
---
## 🔧 Troubleshooting
| Issue | Cause | Solution |
|---|---|---|
| `Connection closed` | Missing `[ai]` extras | Install with `scrapling[ai]` |
| `ModuleNotFoundError: No module named 'click'` | Missing CLI dependencies | `pip install "scrapling[fetchers]"` |
| Cloudflare cannot be bypassed | Need `stealthy_fetch` | Use `stealthy_fetch` instead of `get` |
| x.com returns empty | Need login status | Inject Chrome cookie |
| Timeout | Network/proxy issue | Increase `timeout` / check proxy |
---
## 🙏 Acknowledgements
This project is based on the [Scrapling](https://github.com/D4Vinci/Scrapling) open-source project, thanks to the original author [Karim Shoair (D4Vinci)](https://github.com/D4Vinci) for his outstanding work.
Scrapling is a powerful Web scraping framework:
- GitHub: https://github.com/D4Vinci/Scrapling
- Documentation: https://scrapling.readthedocs.io/
- PyPI: https://pypi.org/project/scrapling/
---
## 📄 License
[MIT License](LICENSE) — Based on Scrapling (BSD License) adaptation
