Content
# 🔍 Search MCP Server
> A one-stop search solution for security research articles from WeChat public accounts, Qi An Xin attack and defense community, and Xian Zhi community, with optional KimiCode universal web search.
## ✨ Features
- 🔎 **Multi-source aggregation** — WeChat public accounts / Qi An Xin attack and defense community / Xian Zhi community, search all security vertical sites with one command
- 🌐 **Universal web search (optional)** — Integrate with KimiCode search/crawling API to supplement content not covered by three vertical sources
- 🛡️ **TLS fingerprint evasion** — Use `curl_cffi` Chrome TLS fingerprint to reduce Sogou anti-scraping and CAPTCHA triggering; Xian Zhi articles go through Playwright to bypass WAF JS challenges
- 🧠 **Intelligent recall** — Local source 0 hits automatically peel off version numbers/noise words, split Chinese-English connected writing and re-search once (e.g., `fastjson1.2.80util` → `fastjson`)
- 🧹 **Result organization** — Cross-source deduplication by URL, abstract blank normalization and preview truncation, optional sorting by date
- 📄 **Article full-text extraction** — Automatically identify source and extract clean text, retain code blocks; fallback to KimiCode crawling for unknown sources
- ⚡ **Concurrent search** — `search_all` query multiple sources simultaneously
## 📦 Installation
```bash
pip install -e .
python -m playwright install chromium # Required for Xian Zhi article retrieval
```
## ⚙️ MCP Configuration
```json
{
"mcpServers": {
"search-mcp": {
"command": "python",
"args": ["-m", "search_mcp"],
"env": {
"KIMI_CODE_API_KEY": "sk-kimi-xxxxxxxx"
}
}
}
}
```
- `command` should point to the Python environment where this package is installed (`pip install -e .`); alternatively, use the console script `search-mcp`.
- `KIMI_CODE_API_KEY`: KimiCode API key (`sk-kimi-...`), required only for `search_web` and `search_all(include_web=True)`. Not related to Kimi/Moonshot open platform's `KIMI_API_KEY`.
## 🖥️ CLI Mode
Start MCP server without subcommands; with `search` / `fetch` subcommands, it's a one-time CLI that reuses the same tool functions as MCP, with consistent results.
```bash
search-mcp search "fastjson1.2.80util" # Default aggregate local three sources (including automatic downgrade recall)
search-mcp search "springutil" --web # Add KimiCode universal web search (requires key)
search-mcp search "XSS" --source butian # Specify single source
search-mcp search "SSRF" --source web --json # Universal web + raw JSON output
search-mcp search "RCE" --sort-by-date # Sort by date in descending order
search-mcp fetch https://xz.aliyun.com/t/xxxx # Retrieve article full text
```
`search` options: `--source {all,weixin,butian,xianzhi,web}`, `--web`, `--page`, `--limit`, `--content`, `--sort-by-date`, `--json`. Each source hit status printed to stderr, results to stdout (for easy piping).
## 🛠️ Tool Overview
| Tool | Parameters | Description |
| ---------------- | --------------------------------------------------- | ------------------------------------------------------------ |
| `search_weixin` | `query`, `page=1` | Search WeChat public account articles (Sogou WeChat search) |
| `search_butian` | `query`, `page=1` | Search Qi An Xin attack and defense community articles |
| `search_xianzhi` | `query`, `page=1` | Search Xian Zhi community articles |
| `search_all` | `query`, `page=1`, `include_web=False`, `sort_by_date=False` | Concurrently search local three sources (with `include_web=True` add universal web search), deduplicate and merge, with `sources` status |
| `search_web` | `query`, `limit=10`, `include_content=False` | KimiCode universal web search (requires key; query routed to api.kimi.com) |
| `fetch_article` | `url` | Retrieve article full text, automatically identify source routing; fallback to KimiCode crawling for unknown sources |
> 💡 Local source (butian/xianzhi/WeChat) site search is literal matching, **version numbers or Chinese-English connected compound words may have 0 hits** — intelligently recalled; for specific versions/CVEs, directly use `search_web`.
### Return Format
```json
{
"results": [
{
"title": "Article Title",
"url": "https://...",
"abstract": "Abstract preview (blank normalized, within 200 characters)",
"source": "weixin",
"author": "Public account name / site name",
"date": "2025-01-01",
"content": "Full text (only with search_web include_content=True)"
}
],
"count": 10,
"sources": { "weixin": 10, "butian": 8, "xianzhi": 0, "web": 4 }
}
```
- Empty fields automatically omitted; `sources` only returned by `search_all` (values are counts or `"error"` to distinguish between "no results" and "source rate-limited/CAPTCHA blocked").
- Cross-source results deduplicated by normalized URL; local native crawling prioritized over universal web search duplicate items; `count` is deduplicated quantity.
## 🔐 Anti-scraping Bypass Principle
| Source | Search | Article Retrieval | Technique |
| ------------ | ------------------ | ---------- | ---------------------------------------------------- |
| WeChat (Sogou) | curl_cffi | curl_cffi | Chrome TLS fingerprint + JS redirect URL splicing parsing |
| Qi An Xin | curl_cffi | curl_cffi | No anti-scraping restrictions |
| Xian Zhi | curl_cffi + CSRF | Playwright | CSRF Token + Alibaba WAF JS Challenge (headless) |
| Universal (Kimi) | KimiCode HTTP API | KimiCode | Official search/fetch API (requires key) |
> TLS fingerprint used to **evade/reduce** Sogou anti-scraping triggering, not CAPTCHA auto-solving; if blocked, corresponding source returns empty results.
## 🧩 Extend New Sources
Create a new scraper in `search_mcp/scrapers/`, three steps:
```python
# search_mcp/scrapers/your_source.py
from .base import BaseScraper
from ..types import SearchResult
class YourScraper(BaseScraper):
name = "your_source"
async def search(self, query: str, page: int = 1) -> list[SearchResult]:
# Implement search logic; use await self._fetch(url) to send requests
...
async def fetch_article(self, url: str) -> str:
# Implement article extraction logic
...
```
Then register in `scrapers/__init__.py` and `server.py`.
## 📁 Project Structure
```
search_mcp/
├── __init__.py
├── __main__.py # Entry point
├── server.py # MCP tool registration + deduplication/cleaning/recall
├── types.py # SearchResult data model
└── scrapers/
├── base.py # BaseScraper (curl_cffi AsyncSession)
├── weixin.py # WeChat search
├── butian.py # Qi An Xin attack and defense community
├── xianzhi.py # Xian Zhi community
└── kimi.py # KimiCode universal web search / crawling
```
