Content
# 🔥 MediaCrawler MCP - Media Platform Crawler
> This project is based on [NanmiCoder/MediaCrawler](https://github.com/NanmiCoder/MediaCrawler) with added support for MCP Server.
A MCP service that supports multi-platform media data crawling, which can be directly called in AI editors like Claude Code, Cursor.
## ✨ Supported Platforms
| Platform | Code | Features |
|------|------|------|
| Xiaohongshu | xhs | Keyword search, comment collection, sub-comment collection |
| Douyin | dy | Keyword search, comment collection, sub-comment collection |
| Kuaishou | ks | Keyword search, comment collection, sub-comment collection |
| Bilibili | bili | Keyword search, comment collection, sub-comment collection |
| Weibo | wb | Keyword search, comment collection, sub-comment collection |
| Baidu Tieba | tieba | Keyword search, comment collection, sub-comment collection |
| Zhihu | zhihu | Keyword search, comment collection, sub-comment collection |
| Toutiao | toutiao | Keyword search, comment collection, sub-comment collection |
## 📋 Environment Installation
### 1. Install uv (Recommended)
```bash
# macOS/Linux
curl -LsSf https://astral.sh/uv/install.sh | sh
# Windows
powershell -c "irm https://astral.sh/uv/install.ps1 | iex"
```
### 2. Install Dependencies
```bash
# Enter project directory
cd mediacrawlermcp
# Install Python dependencies
uv sync
# Install browser driver (will download required Node.js runtime)
uv run playwright install chromium
# Install Git hooks (automatically sync dependencies after pulling code, execute once)
# macOS / Linux
bash setup-hooks.sh
# Windows PowerShell
.\setup-hooks.ps1
```
> **Note**: If you need to develop/build documentation site locally, you need to install Node.js >= 16.0.0.
### 3. Team Collaboration: Automatic Dependency Sync
The project provides a Git post-merge hook that automatically executes `uv sync` when `uv.lock` changes, eliminating the need for manual synchronization.
**Initial setup (once per person):**
```bash
# macOS / Linux
bash setup-hooks.sh
# Windows PowerShell
.\setup-hooks.ps1
```
After setting up, if `uv.lock` changes during `git pull`, dependencies will be automatically synced; otherwise, no operation will be performed.
> Different mirror sources (Tsinghua, Alibaba, etc.) do not affect `uv.lock` content; teams can configure their own mirrors and lock versions consistently.
## 🤖 MCP Configuration
### Claude Code / Cursor Configuration
#### macOS / Linux
Create `.claude/settings.json`:
```json
{
"mcpServers": {
"mediacrawler": {
"command": "uv",
"args": ["run", "python", "mcp_server.py"],
"env": {
"PYTHONPATH": "."
}
}
}
}
```
#### Windows (Recommended)
Use cmd to switch directory and execute:
```json
{
"mcpServers": {
"mediacrawler": {
"command": "cmd",
"args": [
"/c",
"cd /d D:\\mcp_work\\mediacrawlermcp && uv run python mcp_server.py"
]
}
}
}
```
#### Windows (Alternative: Directly call virtual environment Python)
If `uv` command has issues, you can directly call virtual environment Python (note: must cd to project directory):
```json
{
"mcpServers": {
"mediacrawler": {
"command": "cmd",
"args": [
"/c",
"cd /d D:\\mcp_work\\mediacrawlermcp && .venv\\Scripts\\python.exe mcp_server.py"
]
}
}
}
```
**Optional environment variables (all optional, not configuring will disable AI report mode):**
| Variable | Description |
|------|------|
| `ANTHROPIC_API_KEY` | Anthropic API Key (for AI report mode) |
| `ANTHROPIC_BASE_URL` | Anthropic API address (compatible endpoints allowed) |
| `ANTHROPIC_DEFAULT_SONNET_MODEL` | Optional, specify model name (e.g., `claude-sonnet-4-6`), automatically injected when running with Claude Code, can be manually configured when running independently |
**Configuration methods (choose one, priority from high to low):**
1. **MCP `env` field** — Add `"env": {"ANTHROPIC_API_KEY": "sk-xxx", ...}` to the above JSON.
2. **System environment variables** — Set in the operating system.
3. **`.env` file** — Create `.env` file in project root directory and write variables (will be overridden by the first two methods).
> ⚠️ If an empty string is written in MCP `env` (e.g., `"ANTHROPIC_API_KEY": ""`), it will override the same variable in `.env` file, causing AI mode to fail. Unneeded variables should not be written, and empty strings should not be left.
## 🚀 Usage
After configuration, directly use natural language or structured parameters to crawl content in AI editors.
### 1. crawl_media Tool - Single-Platform Crawling
**Basic usage:**
```python
# Crawl Xiaohongshu posts and comments about "Perfect Diary"
result = await crawl_media(
platform="xhs",
crawler_type="search",
keywords="Perfect Diary",
max_count=50,
is_get_comments=True,
max_comments_count=10
)
```
**Parameter description:**
| Parameter | Required | Description | Optional values |
|------|------|------|--------|
| `platform` | Yes | Platform code | `xhs`, `dy`, `ks`, `bili`, `wb`, `tieba`, `zhihu`, `toutiao` |
| `crawler_type` | No | Crawling type | `search`(default), `detail`, `creator` |
| `keywords` | Yes | Search keywords | arbitrary text |
| `max_count` | No | Crawling quantity | 1-100, default 20 |
| `is_get_comments` | No | Whether to get comments | `true`/`false`, default `false` |
| `is_get_sub_comments` | No | Whether to get sub-comments | `true`/`false`, default `false` |
| `max_comments_count` | No | Primary comment count and sub-comment count per comment | 0-50, default 20 |
| `report_type` | No | Report type | see below, default `sentiment` |
| `report_mode` | No | Report mode | `auto`(default), `ai`, `script` |
| `save_data_option` | No | Data storage method | `""`(no storage), `"db"` |
| `output_path` | No | Report output directory | default `reports` |
**Report types:**
| Type | Code | Description |
|------|------|------|
| Sentiment analysis | `sentiment` | Emotional distribution, positive/negative analysis, sentiment insights |
| Trend analysis | `trend` | Popularity trend, timeline analysis, change prediction |
| Hot topics | `hot_topics` | Popular topic identification, topic clustering, propagation path |
| Keyword analysis | `keyword` | Keyword frequency, associated words, semantic network |
| Volume analysis | `volume` | Volume statistics, peak detection, platform distribution |
| Viral spread | `viral_spread` | Propagation nodes, fission path, KOL influence |
| Influencer analysis | `influencer` | KOL identification, influence assessment, cooperation suggestions |
| Audience analysis | `audience` | User portrait, interest distribution, active time |
| Competitor comparison | `comparison` | Competitor extraction, comparison analysis, pros and cons |
| Risk warning | `risk` | Risk identification, crisis warning, response suggestions |
**Natural language examples:**
```
Search popular posts about "Python programming" on Xiaohongshu
Crawl videos and comments about "Artificial Intelligence" on Bilibili for sentiment analysis
Analyze the propagation trend of "Food exploration" on Douyin
Weibo risk warning report on "New Energy Vehicles"
```
**Return example:**
```json
{
"status": "success",
"platform": "bili",
"platform_name": "Bilibili",
"report_mode": "ai_enhanced",
"keywords": "Artificial Intelligence",
"report_path": "/path/to/reports/Sentiment Analysis Report_xxx.html",
"relative_path": "reports/Sentiment Analysis Report_xxx.html",
"summary": "Sentiment analysis summary...",
"has_ai_config": true,
"verification_samples": [],
"message": "Report generated"
}
```
### 2. crawl_multi_platform Tool - Multi-Platform Crawling
**Crawl multiple platforms simultaneously, generate unified report**, making it convenient to compare and analyze platform differences.
**Basic usage:**
```python
# Crawl data from Bilibili, Douyin, and Xiaohongshu platforms simultaneously
result = await crawl_multi_platform(
platforms=["bili", "dy", "xhs"],
crawler_type="search",
keywords="Artificial Intelligence",
max_count=20,
is_get_comments=True,
max_comments_count=10
)
```
**Multi-platform report features:**
- One HTML report contains all platform data
- Displays platform content distribution ratio
- Cross-platform hot word unified analysis
- Platform sentiment tendency comparison
- Platform content strategy suggestions
- Single platform failure will not interrupt the entire operation
**Parameter description:**
| Parameter | Required | Description | Optional values |
|------|------|------|--------|
| `platforms` | Yes | Platform code list | `["xhs"]`, `["bili", "dy"]`, etc. |
| `crawler_type` | No | Crawling type | `search`(default), `detail`, `creator` |
| `keywords` | Yes | Search keywords | arbitrary text |
| `max_count` | No | Crawling quantity per platform | 1-100, default 20 |
| `is_get_comments` | No | Whether to get comments | `true`/`false`, default `false` |
| `is_get_sub_comments` | No | Whether to get sub-comments | `true`/`false`, default `false` |
| `max_comments_count` | No | Primary comment count and sub-comment count per comment | 0-50, default 20 |
| `report_type` | No | Report type | see above, default `sentiment` |
| `report_mode` | No | Report mode | `auto`(default), `ai`, `script` |
| `output_path` | No | Report output directory | default `reports` |
**Return example:**
```json
{
"status": "success",
"platforms": ["bili", "dy", "xhs"],
"platform_names": ["Bilibili", "Douyin", "Xiaohongshu"],
"report_mode": "ai_enhanced",
"keywords": "Artificial Intelligence",
"total_items": 60,
"platform_breakdown": {"xhs": 20, "dy": 20, "bili": 20},
"report_path": "/path/to/reports/Multi-Platform_Artificial Intelligence_Sentiment Analysis Report_xxx.html",
"relative_path": "reports/Multi-Platform_Artificial Intelligence_Sentiment Analysis Report_xxx.html",
"summary": "Multi-platform comprehensive analysis summary...",
"verification_samples": {},
"message": "Multi-platform sentiment analysis report generated"
}
```
**Natural language examples:**
```
Crawl Bilibili, Douyin, and Xiaohongshu platforms for "National Tide Brand" content comparison analysis
Analyze Weibo and Zhihu discussions on "New Energy Vehicles" simultaneously
Use script mode to generate multi-platform comparison report
```
### 3. Auxiliary Tools
| Tool name | Function | Usage |
|--------|------|------|
| `get_platforms` | Get supported platform list | Display all crawlable platform codes, names, and descriptions |
| `get_crawler_types` | Get supported crawling types | Display available crawling type descriptions |
### Report Mode Description
Automatically generate sentiment analysis reports after crawling:
| Mode | Description | Applicable scenarios |
|------|------|----------|
| `auto`(default) | Automatic detection: use AI if LLM configured, otherwise use script | No need to care about underlying implementation, use with one click |
| `ai` | Forced use of LLM to dynamically design reports based on data features | Need flexible, personalized analysis |
| `script` | Forced use of preset templates to generate unified format reports | Need fixed format, batch comparison, offline use |
**⚠️ AI mode requires LLM API configuration**
Before using AI mode, configure one of the following environment variables:
**Method 1: OpenAI compatible format (supports DeepSeek, Silicon Flow, etc.)**
```bash
# .env file or environment variable
OPENAI_API_KEY=your_api_key
OPENAI_BASE_URL=https://api.deepseek.com/v1 # or other compatible endpoints
LLM_MODEL=deepseek-chat # model name
```
**Method 2: Anthropic native API**
```bash
ANTHROPIC_API_KEY=your_api_key
ANTHROPIC_BASE_URL=https://api.anthropic.com
```
**Common model configuration reference:**
| Service provider | OPENAI_BASE_URL | LLM_MODEL |
|--------|-----------------|-----------|
| DeepSeek | `https://api.deepseek.com/v1` | `deepseek-chat` |
| Silicon Flow | `https://api.siliconflow.cn/v1` | `Qwen/Qwen2.5-72B-Instruct` |
| Alibaba Cloud Baizhan | `https://dashscope.aliyuncs.com/compatible-mode/v1` | `qwen-max` |
| Self-deployed | `http://localhost:8000/v1` | your model name |
> **Note**: AI mode will call LLM API to generate reports, which may incur API call fees. For offline use, switch to `script` mode.
### CDP Browser Mode
Supports connecting to user real browser through Chrome DevTools Protocol to improve anti-detection capabilities.
**Configuration items (`config/base_config.py`):**
| Configuration | Default value | Description |
|------|--------|------|
| `ENABLE_CDP_MODE` | `True` | Enable CDP mode |
| `CDP_DEBUG_PORT` | `9222` | CDP debug port |
| `CUSTOM_BROWSER_PATH` | `""` | Custom browser path |
| `CDP_HEADLESS` | `False` | Whether to use headless mode |
| `BROWSER_LAUNCH_TIMEOUT` | `60` | Browser launch timeout (seconds) |
| `AUTO_CLOSE_BROWSER` | `True` | Automatically close browser |
### Complete Usage Example
**Example 1: Single-platform sentiment monitoring**
```python
result = await crawl_media(
platform="xhs",
keywords="Perfect Diary",
max_count=50,
is_get_comments=True
)
```
**Example 2: Multi-platform comparative analysis**
```python
result = await crawl_multi_platform(
platforms=["bili", "dy", "xhs"],
keywords="National Tide Brand",
max_count=30,
is_get_comments=True
)
```
**Example 3: Risk warning report**
```python
result = await crawl_media(
platform="wb",
keywords="Brand Crisis",
report_type="risk",
is_get_comments=True
)
```
**Example 4: Static report (fixed format)**
```python
result = await crawl_media(
platform="bili",
keywords="Artificial Intelligence",
report_mode="script"
)
```
## 📁 Project Structure
```
.
├── mcp_server.py # MCP service entry (4 tools)
├── mcp_core/ # MCP layer
│ └── mcp_adapter.py # Crawler adapter, bridges MCP and underlying crawler
├── reporting/ # Report generation layer
│ ├── report_generator.py # Script report generator (11 report types)
│ ├── ai_report_generator.py # AI report data preparation and prompt construction
│ ├── llm_report_generator.py # LLM API call and HTML report generation
│ └── auto_field_detector.py # Automatic field mapping (eliminates hard-coded dependencies)
├── tools/cdp_browser.py # CDP browser manager
├── media_platform/ # Platform crawler implementation
│ ├── xhs/ # Xiaohongshu
│ ├── douyin/ # Douyin
│ ├── kuaishou/ # Kuaishou
│ ├── bilibili/ # Bilibili
│ ├── weibo/ # Weibo
│ ├── tieba/ # Baidu Tieba
│ ├── zhihu/ # Zhihu
│ └── toutiao/ # Toutiao
├── config/ # Configuration files
├── store/ # Data storage
├── original_data/ # Crawled raw data (JSON)
├── reports/ # Generated report output directory
└── pyproject.toml # Project dependencies
```
## ⚠️ Disclaimer
This project is for learning and research purposes only, prohibited for commercial use and illegal activities. Using this project implies that you agree to bear all related responsibilities.
Original project: https://github.com/NanmiCoder/MediaCrawler
