Content
# Tool List
Based on the **FastMCP** framework, a WeChat official account article crawling system that allows AI agents to directly access and analyze WeChat official account content. By implementing the MCP (Model Context Protocol) standard protocol, seamless integration between AI agents and Selenium crawlers is achieved.
## 🎯 Project Background
When using AI platforms or agents, we found that agents cannot directly access WeChat official account article content. To solve this problem, we developed this MCP-based crawler service that allows AI agents to retrieve and analyze WeChat official account content.
## ✨ Core Features
- 🤖 **FastMCP Framework** - Based on FastMCP's advanced encapsulation, simplifying MCP server development
- 🕷️ **Intelligent Crawler** - Using Selenium to automate browser, supporting dynamic content capture
- 🖼️ **Image Processing** - Automatically download article images and convert them to local files
- 📊 **Content Analysis** - Providing article statistics, keyword extraction, and other analysis functions
- 🔌 **Standard Protocol** - Fully compatible with MCP 1.0+ specifications, supporting stdio communication
- 🎯 **AI Integration** - Seamlessly integrated with Claude Desktop, ChatGPT, and other AI agents
- 💻 **Multiple Interfaces** - Providing Python API and interactive command-line interface
- 🧪 **Dependency Check** - Automatically detecting and installing missing dependency packages
- 📁 **Data Storage** - Supporting multiple format file saving (JSON, TXT)
- 🚀 **Performance Optimization** - Memory and CPU usage optimization
## 🏗️ System Architecture
```mermaid
graph TB
subgraph "AI Agent Layer"
A[Claude Desktop]
B[ChatGPT]
C[Other AI Agents]
end
subgraph "MCP Protocol Layer"
D[MCP Client]
E[stdio Communication]
F[MCP Server<br/>FastMCP]
end
subgraph "Crawler Engine Layer"
G[Selenium WebDriver]
H[Chrome Browser]
I[Image Downloader]
end
subgraph "Data Storage Layer"
J[JSON File]
K[TXT File]
L[Image File]
end
A --> D
B --> D
C --> D
D <--> E
E <--> F
F --> G
G --> H
F --> I
G --> J
G --> K
I --> L
```
### 🔧 Core Components
#### 1. FastMCP Server (`src/mcp_weixin_spider/server.py`)
- Advanced encapsulation based on the FastMCP framework
- Providing three core tools: article crawling, content analysis, and statistical information
- Singleton pattern managing Selenium crawler instances
- Complete error handling and parameter verification
- Supporting modular startup and command-line script invocation
#### 2. MCP Standard Client (`src/mcp_weixin_spider/client.py`)
- Standard MCP protocol client implementation
- Asynchronous communication and session management
- Interactive command-line interface
- Python API interface
- Supporting command-line script startup
#### 3. Module Entry (`src/mcp_weixin_spider/__main__.py`)
- Unified module startup entry
- Supporting server and client operation modes
- Providing clear command-line help information
#### 4. Configuration Management (`src/mcp_weixin_spider/config.py`)
- TOML format configuration file support
- Environment variable coverage mechanism
- Type-safe configuration access
#### 5. Selenium Crawler Engine (`src/mcp_weixin_spider/spider.py`)
- Chrome browser automation control
- Intelligent ChromeDriver management (automatic installation and path detection)
- Anti-crawler mechanism processing
- Image download and format conversion
- Multi-format file saving
- Memory and performance optimization
#### 6. Dependency Checker (`src/mcp_weixin_spider/spider.py`)
- Automatic system dependency detection
- Providing detailed dependency missing information
- Supporting one-click installation of missing dependencies
## 📁 Project Structure
```
MCPWeChatOfficialAccounts/
├── src/
│ └── mcp_weixin_spider/
│ ├── __init__.py # Package initialization file
│ ├── __main__.py # Module entry point
│ ├── client.py # MCP client implementation
│ ├── config.py # Configuration management
│ ├── exceptions.py # Exception definition
│ ├── main.py # Main function
│ ├── server.py # FastMCP server implementation
│ └── spider.py # Selenium crawler engine
├── weixin_spider.py # Crawler test script
├── config.toml # Main configuration file
├── config.toml.example # Configuration file example
├── LICENSE # License file
├── pyproject.toml # Project metadata and dependency management
└── README.md # Project documentation
```
## 🚀 Quick Start
### 📋 Environment Requirements
- **Python**: 3.10+
- **Browser**: Chrome/Chromium (automatically managing ChromeDriver)
- **System**: macOS/Windows/Linux
### 📦 Installation Steps
### Traditional Installation (using source code)
```bash
# 1. Clone the project
git clone https://github.com/example/mcp-weixin-spider.git
cd MCPWeChatOfficialAccounts
# 2. Install dependencies
pip install .
# 3. Development mode installation (optional, for developers)
pip install -e .[dev]
```
### Installation using pip
```bash
# Install the latest version
pip install mcp-weixin-spider
```
### ⚙️ Configuration Management
The project uses TOML format configuration files, supporting configuration through configuration files and environment variables.
#### Configuration File
Configuration file description:
```toml
[spider]
headless = true # Whether to run the browser in headless mode
wait_time = 10 # Page waiting time (seconds)
download_images = true # Whether to download images in articles
browser = "chrome" # Browser type, supporting 'chrome' and 'edge'
chrome_driver_path = "" # ChromeDriver path (optional, automatically managed)
edge_driver_path = "" # EdgeDriver path (optional, automatically managed)
articles_dir = ".temp" # Article saving directory
images_dir = ".images" # Image saving directory
[mcp]
server_name = "mcp-weixin-spider" # MCP server name
transport = "stdio" # Transport method (stdio or tcp)
debug = false # Whether to enable debug mode
[log]
level = "INFO" # Log level (DEBUG, INFO, WARNING, ERROR, CRITICAL)
format = "%(asctime)s - %(name)s - %(levelname)s - %(message)s" # Log format
file = "" # Log file path (optional, output to console if left blank)
```
#### Environment Variables
Supporting configuration override through environment variables:
| Environment Variable | Corresponding Configuration Item | Description |
| -------------------- | --------------------------- | ------------------ |
| HEADLESS | spider.headless | Whether to use headless mode |
| DOWNLOAD\_IMAGES | spider.download\_images | Whether to download images |
| WAIT\_TIME | spider.wait\_time | Page waiting time |
| BROWSER | spider.browser | Browser type (chrome or edge) |
| CHROME\_DRIVER\_PATH | spider.chrome\_driver\_path | ChromeDriver path (optional) |
| EDGE\_DRIVER\_PATH | spider.edge\_driver\_path | EdgeDriver path (optional) |
| ARTICLES\_DIR | spider.articles\_dir | Article saving directory |
| IMAGES\_DIR | spider.images\_dir | Image saving directory |
| MCP\_SERVER\_NAME | mcp.server\_name | MCP server name |
| MCP\_TRANSPORT | mcp.transport | Transport method (stdio or tcp) |
| MCP\_DEBUG | mcp.debug | Whether to enable debug mode |
| LOG\_LEVEL | log.level | Log level |
| LOG\_FILE | log.file | Log file path |
### 🎮 Startup Method
#### Using Command-Line Script (recommended)
```bash
# Start MCP server (default mode)
mcp-weixin-spider
```
#### Using Modular Startup
```bash
# Start MCP server (default mode)
python -m mcp_weixin_spider
# Start MCP server (explicitly specify server mode)
python -m mcp_weixin_spider server
# Start interactive client
python -m mcp_weixin_spider client
```
#### Testing Crawler Function
```bash
# Run crawler test script
python weixin_spider.py
```
## 🛠️ MCP Tool Interface
### Tool List
| Tool Name | Function Description | Parameters | Return Value |
| ---------------------- | --------- | -------------------------------------------------------------- | --------------- |
| `crawl_weixin_article` | Crawl WeChat official account article | `url`: Article URL`download_images`: Whether to download images`custom_filename`: Custom filename | JSON object containing article content |
| `analyze_article` | Analyze article content | `article_content`: Article content | Analysis result (keywords, statistics, etc.) |
| `get_article_stats` | Get article statistics | `article_content`: Article content | Article statistics data |
## 👨💻 Development Guide
### Development Environment Setup
```bash
# Clone the project
git clone https://github.com/example/mcp-weixin-spider.git
cd MCPWeChatOfficialAccounts
# Install development dependencies
pip install -e .[dev]
# Run code formatting
black src/ weixin_spider.py
# Run code check
flake8 src/ weixin_spider.py
# Run type check
mypy src/
```
### Project Build
```bash
# Build the project
python -m build
# Generated package files will be located in the dist/ directory
```
### Test
```bash
# Run crawler test
python weixin_spider.py
```
### Contribution Guide
1. Follow PEP 8 coding specifications
2. Use Black for code formatting
3. Add appropriate type annotations
4. Ensure all code passes flake8 check
5. Write clear commit messages
6. Run all check commands before submission
## ⚠️ Precautions
1. **Usage Frequency**: Please do not use the crawler too frequently to avoid putting pressure on the WeChat server
2. **Legality**: Please ensure that your crawling behavior complies with relevant laws and regulations and WeChat public platform usage terms
3. **Chrome Version**: It is recommended to use the latest version of Chrome browser for optimal compatibility
4. **Memory Usage**: Long-term operation may consume more memory, and it is recommended to restart the service regularly
5. **Network Environment**: Please ensure a stable network environment to avoid crawling failure due to network issues
## 📄 License
Apache License 2.0
## 📝 Update Log
### v0.1.0 (2026-03-13)
- Initial version release
- Implemented WeChat official account article crawling function
- Supported MCP protocol
- Supported image download and saving
- Provided AI agent integration interface
