Content
# wechat-to-kb
<!-- mcp-name: io.github.careycao/wechat-to-kb -->
> Unify WeChat public accounts, web pages, videos, Xiaohongshu, and RSS feeds into a local knowledge base for AI retrieval and Q&A.
---
## Why Build This
Information is not lacking, but it's scattered and hard to reuse.
Public account articles are forgotten after reading, and collections are hard to find; Bilibili videos and Xiaohongshu notes disappear after watching; RSS subscriptions pile up with dozens of sources that can't be read. This is a problem with new content.
Old content is even harder to handle: third-party cloud notes like Youdao have accumulated thousands of articles over the years, but are locked in a closed platform and unreadable by AI; local hard drives are filled with PDFs, PPTXs, and DOCXs - training materials, conference shares, and industry reports that have never been systematically organized.
**On the surface, knowledge is accumulating, but in reality, only a small part of it can be called upon by AI.**
The goal of wechat-to-kb is to connect all of these: WeChat public accounts, web pages, videos (Bilibili/YouTube/Xiaohongshu), RSS subscriptions, Youdao cloud notes, and local PDFs/PPTXs/DOCXs, and unify them into a local knowledge base, making it a structured, searchable, and AI-Q&A-capable knowledge asset.
Tools and models are rapidly iterating, but **releasing your valuable historical accumulation and letting AI read it stably** is a fundamental personal construction in the AI era.
---
## Contains Ten Modules
| Module | Function |
|---|---|
| `run.sh` | Top-level unified entrance, chat tool only needs to connect to this, and internally automatically distribute to WeChat/public account/video/webpage |
| `mcp_server` | **MCP Server**, allowing any MCP client like Claude Desktop/Cursor to directly call without a terminal |
| `wechat_collector` | WeChat public account article collector, managing WeChat login status, and retaining comment PoC capability |
| `web_collector` | Ordinary webpage collector, automatically routing to the corresponding knowledge base |
| `common` | Shared knowledge base configuration, routing, storage, indexing, and text processing layer for the entire repository |
| `video_collector` | Video-to-text, supporting Bilibili, YouTube, Xiaohongshu video (yt-dlp + subtitle extraction) |
| `xhs_collector` | Xiaohongshu favorite batch import |
| `rss_daily` | RSS subscription aggregation, WeChat public account article automatic archiving |
| `platform_collector` | **Platform Radar**: automatically crawling public hotlists (HackerNews, GitHub Trending, etc.), AI summary generates daily Inbox summary, not written to knowledge base |
| `tools/import_local_docs.py` | Local PDF batch import (MarkItDown extraction + Claude value assessment), entrance script `run_import_local_docs.sh` |
All content is uniformly stored as **local text files**, classified and managed by knowledge base, and can be directly connected to any AI tool that supports local files (OpenClaw, Cursor, Obsidian, RAG, etc.).
### LLM Enhanced Import (enabled by default)
Each article is processed with two steps of CoT (Chain of Thought) when imported, generating structured frontmatter:
```yaml
---
title: Article Title
url: https://mp.weixin.qq.com/s/xxxxx # Automatically removed UTM/tracking parameters
date: 2026-05-01
summary: 3 sentences of core viewpoint summary
concepts:
- Core concept A
- Core concept B
related:
- KB internal related article title 1
- KB internal related article title 2
---
```
The `related` field is compatible with Obsidian Graph View, and you can see the association between articles directly in the knowledge graph. If you enable `KB_WIKILINKS=1`, a related article block with `[[article title]]` syntax will be appended to the end of the text.
**Authentication**: Same as local PDF value assessment, defaulting to `claude` CLI (Claude Code) on this machine, no additional API Key required.
**Related environment variables:**
| Variable | Default value | Description |
|------|--------|------|
| `KB_ENRICH` | `1` (run.sh)/need manual setting (MCP) | Enable two-step CoT enhancement |
| `KB_ENRICH_MODEL` | `claude-haiku-4-5-20251001` | Specify the model used for enhancement, e.g., `claude-sonnet-4-6` |
| `KB_WIKILINKS` | `0` | Enable appending `[[wikilinks]]` format related article blocks at the end of the text |
| `KB_ROUTE_LLM` | `1` (run.sh)/need manual setting (MCP) | Enable LLM intelligent routing, using LLM to determine the target knowledge base when the confidence is low |
| `KB_ROUTE_MODEL` | `claude-haiku-4-5-20251001` | Specify the model used for routing |
---
## MCP Server (Recommended for new users)
In any AI tool that supports MCP, such as Claude Desktop, Cursor, Cowork, directly dialogue and save content without configuring a terminal tool.
### Installation
```bash
# Method 1: uvx (no clone required, recommended)
uvx --from 'wechat-to-kb[mcp]' wechat-to-kb-mcp
# Method 2: For users who have cloned this project
pip install -e ".[mcp]"
```
### Configure Claude Desktop
Edit `~/Library/Application Support/Claude/claude_desktop_config.json`:
```json
{
"mcpServers": {
"wechat-to-kb": {
"command": "uvx",
"args": ["--from", "wechat-to-kb[mcp]", "wechat-to-kb-mcp"],
"env": {
"KB_ROOT": "/Users/username/knowledge_base",
"KB_NON_INTERACTIVE": "1",
"KB_ENRICH": "1",
"KB_ROUTE_LLM": "1",
"KB_ROUTE_MODEL": "deepseek-v4-flash",
"DEEPSEEK_API_KEY": "your DeepSeek Key (optional for platform_collector summary)"
}
}
}
}
```
### Configure Codex
Edit `~/.codex/config.toml` or add and manually complete environment variables after `codex mcp add`:
```toml
[[mcp_servers]]
name = "wechat-to-kb"
command = "uvx"
args = ["--from", "wechat-to-kb[mcp]", "wechat-to-kb-mcp"]
[mcp_servers.env]
KB_ROOT = "/Users/username/knowledge_base"
KB_NON_INTERACTIVE = "1"
KB_ENRICH = "1"
KB_ROUTE_LLM = "1"
KB_ROUTE_MODEL = "deepseek-v4-flash"
DEEPSEEK_API_KEY = "your DeepSeek Key"
```
After configuration, use `codex mcp get wechat-to-kb` to verify if the variables take effect; open a new session to read the latest configuration.
> **Note**: `DEEPSEEK_API_KEY` is stored in plain text in `~/.codex/config.toml`. Please do not submit this file to the code repository.
After restarting, directly say in the dialogue:
```
Help me save this article to the knowledge base: https://mp.weixin.qq.com/s/xxxxx
List the categories in my knowledge base
Help me import ~/Downloads/industry report.pdf to the knowledge base
```
Provided tools: `fetch_url` (read-only)/`save_url` / `save_urls_batch` / `import_local_file` / `list_knowledge_bases` / `rebuild_index`
For detailed instructions, see [mcp_server/README.md](mcp_server/README.md).
---
## Usage
> Convention: One-time temporary scripts/temporary verification codes are uniformly placed in `scratch/`, which is added to `.gitignore` and will not be uploaded to GitHub.
### Method 1: MCP Server (Recommended, see above section)
Claude Desktop/Cursor and other tools that support MCP directly call without a terminal, see the "MCP Server" section above.
### Method 2: In AI Assistant Dialogue (OpenClaw/Feishu)
No need to open a terminal. In any AI assistant that accesses the `run.sh` root directory, directly say a sentence to save:
**OpenClaw/Cursor dialogue:**
```
Help me save this article to the knowledge base https://mp.weixin.qq.com/s/xxxxxx
```
**Feishu robot:**
Share the WeChat public account article link with the AI Bot, say "Save to knowledge base", and the Bot will automatically execute and reply with the save result:
The AI assistant will automatically judge the article category, select the corresponding knowledge base, and return the save result (title, category, core keywords).
**Principle**: The AI assistant directly calls `run.sh` through the exec tool without terminal intervention. In a non-TTY environment (AI background execution), automatically select the knowledge base with the highest score, without manual confirmation.
#### Configure AI Assistant
In your AI assistant's tool description file (e.g., OpenClaw's `TOOLS.md`, Cursor's `AGENTS.md`), add the following entry:
```markdown
**wechat-to-kb (Unified link storage)**
- When the user says "Save to knowledge base <URL>", directly exec and execute:
`/bin/bash ~/path/to/wechat-to-kb/run.sh "<URL>"`
- The script will automatically distribute: WeChat public account -> `wechat_collector`; video station -> `video_collector`; other webpages -> `web_collector`
- Specify knowledge base: add `--kb ai` / `engineering` / `management` / `pm`
- Batch save: `run.sh -f urls.txt`
- No TTY automatically non-interactive, no need to add `-n`
- If facing OpenClaw, it is recommended to return the script stdout to the user without rewriting
```
Replace `~/path/to/wechat-to-kb` with your actual installation path.
Current recommended layout:
- Real repository: `~/DevProjects/wechat-to-kb`
- OpenClaw entrance: `~/.openclaw/wechat-to-kb` (soft link to the real repository)
---
### Method 3: Terminal Command Line
```bash
cd ~/DevProjects/wechat-to-kb
./run.sh "https://mp.weixin.qq.com/s/xxxxxx"
./run.sh "https://example.com/article"
./run.sh "https://www.bilibili.com/video/BVxxxxx"
# Batch
./run.sh -f urls.txt
# Skip LLM enhancement (when network is poor or batch import)
./run.sh --no-enrich "https://mp.weixin.qq.com/s/xxxxx"
# Skip LLM routing and use keyword matching only (token-saving / poor network)
./run.sh --no-llm-route "https://mp.weixin.qq.com/s/xxxxx"
# Force coverage of existing articles (if you need to re-enhance old articles)
./run.sh --no-skip "https://mp.weixin.qq.com/s/xxxxx"
# Delete imported articles (clear files and index records, search in all knowledge bases)
./run.sh --delete "https://mp.weixin.qq.com/s/xxxxx"
# Try to grab comments only for WeChat public accounts
./run.sh --comments "https://mp.weixin.qq.com/s/xxxxx"
```
Explanation:
- The top-level entrance will automatically distribute to `wechat_collector` / `video_collector` / `web_collector`
- This is the most suitable entrance for chat tool configuration, and subsequent internal structure adjustments will not affect external calls
- Default enable LLM two-step CoT enhancement, imported `.md` files will automatically generate `summary`, `concepts`, `related` fields; add `--no-enrich` to skip
- Default enable LLM routing (`KB_ROUTE_LLM=1`), when the classification confidence is low, automatically use LLM to make a judgment to avoid misclassification; add `--no-llm-route` to fall back to pure keyword matching
### wechat_collector (WeChat Public Account)
```bash
cd wechat_collector
# Save a single article
./run.sh "https://mp.weixin.qq.com/s/xxxxx"
# Try to grab comments from WeChat public accounts (PoC)
./run.sh --comments "https://mp.weixin.qq.com/s/xxxxx"
# Specify knowledge base
./run.sh --kb ai "https://..."
# Batch import (urls.txt one link per line)
./run.sh -f urls.txt
# Rebuild index
./run.sh --reindex
```
For detailed instructions, see [wechat_collector/USAGE.md](wechat_collector/USAGE.md).
Explanation:
- `--comments` only works for WeChat public accounts and relies on saved WeChat login status.
- Comments will be appended to the end of the text and written to the knowledge base for subsequent unified retrieval.
- Current PoC mode, comment grabbing failure will not affect article saving.
### web_collector (Ordinary Webpage)
```bash
cd web_collector
./run.sh "https://example.com/article"
./run.sh --kb engineering "https://example.com/article"
./run.sh -f urls.txt
./run.sh --reindex
```
For detailed instructions, see [web_collector/USAGE.md](web_collector/USAGE.md).
### video_collector (Video to Text)
```bash
cd video_collector
# First use: login to Bilibili to get cookies
./run.sh --login
# Save video (extract subtitles/introduction)
./run.sh "https://www.bilibili.com/video/BVxxxxx"
./run.sh "https://www.youtube.com/watch?v=xxxxx"
```
For detailed instructions, see [video_collector/README.md](video_collector/README.md).
### xhs_collector (Xiaohongshu Favorites)
```bash
cd xhs_collector
./run.sh
```
### rss_daily (RSS Subscription Daily)
```bash
cp rss_daily/rss_config.example.yaml rss_daily/config.yaml
# Edit config.yaml and fill in your RSS subscription sources
cd rss_daily && ./run.sh
```
### platform_collector (Platform Radar + Daily Summary)
**Positioning**: Information radar, not a knowledge base collector. Hotlist content has strong timeliness, and direct storage will only increase noise; the correct usage of this module is to run it once a day, use AI to aggregate summaries, generate a "Today's Technology Dynamics" and put it into `Inbox/` to read and go, and then manually store the content that is really worth leaving.
The architecture is divided into two layers:
- `autocli_fetcher.py` — Call the direct connection public API to grab the hotlist (HackerNews Firebase API, GitHub Search API), with multiple interface fallback and SSL downgrade
- `digest_builder.py` — Call LLM (default DeepSeek) to make aggregated summaries on the grabbed results, and output structured Markdown
```bash
cd platform_collector
# Generate today's summary (grab HN + GitHub → DeepSeek summary → write to Inbox/daily_digest/)
python digest_builder.py
# Dry-run (only print, no write file)
python digest_builder.py --dry-run
# Skip AI summary, only output the original hotlist
python digest_builder.py --no-ai
# Only grab and write to KB _stage/ (no summary, used for archive scenarios)
python platform_collector.py hackernews_hot github_trending --dry-run
# List all available tasks
python platform_collector.py --list
```
**Configure DeepSeek API Key** (summary function depends on):
```bash
echo 'DEEPSEEK_API_KEY=your_key' >> ~/DevProjects/wechat-to-kb/.env
```
**Network Description**: Direct connection to API (Firebase, GitHub) requires a proxy to access the international network. Domestic aggregation APIs such as Zhihu have low stability and have been removed from the default tasks. If you need to use it in a proxy environment, ensure that the `HTTPS_PROXY` environment variable is set.
Output file location: `~/knowledge_base/Inbox/daily_digest/YYYY-MM-DD.md`
---
### tools/import_local_docs.py (Local PDF Bulk Import)
Batch import scattered PDFs (training materials, industry reports, historical documents) into the knowledge base, doing three things:
1. **MarkItDown extract text**, and calculate `parse_quality` (characters per page / Chinese proportion / garbled rate)
2. **Claude value evaluation** (default enabled): score in four dimensions (topic_decay / ai_displacement / timelessness / personal_relevance), only `verdict=keep` really write summary cards into the library; boundary entries `verdict=review` original file archive but wait for manual confirmation; `low-value` only archive no index
3. **Unified archive + routing**: original files into `~/knowledge_base/Archive/LocalDocs/imported/pdf/`, summary cards are divided into corresponding KB / classification according to `common/kb_routing.py`
```bash
# Default run, automatically enable value evaluation
./run_import_local_docs.sh --source ~/Documents/PDFs
# Only preview (no write file)
./run_import_local_docs.sh --source ~/Documents/PDFs --dry-run
# After reading the report, copy the hash8 from the report to confirm the review entries
./run_import_local_docs.sh --source ~/Documents/PDFs \
--force-include-hash 3b1b52d7,7418dd5d
# Close value evaluation (back to "all files are imported")
./run_import_local_docs.sh --source ~/Documents/PDFs --no-value-check
```
**Authentication**: Default use of this machine `claude` CLI (Claude Code) login status, **no API Key required**; if Claude Code is not installed, fallback to `ANTHROPIC_API_KEY`.
**Report**: Default output to `~/knowledge_base/Archive/LocalDocs/reports/` (file name contains date and source directory name), divided into four sections (imported / to be confirmed / low value / deduplicated).
**Field query**: The first run will automatically generate frontmatter field descriptions in `~/knowledge_base/Archive/LocalDocs/README.md`, convenient for nearby query in KB. Complete design see `Designs/20260419-local-pdf-import-design.md` (v1.2).
---
## Storage Structure
All content is stored by knowledge base + classification directory, each article retains `.md` (Markdown, for AI retrieval and Obsidian view) and `.html` (original archive) two files, root directory automatically generates `README.md` index:
```
~/knowledge_base/
├── AI_KnowBase/
│ ├── README.md ← Automatically generated article index (title, summary, keywords)
│ ├── purpose.md ← Knowledge base purpose statement (automatically created during initialization, can be manually completed)
│ ├── _url_index.json ← URL deduplication index (normalized URL → title/path/date)
│ ├── 01-Strategy and Framework/
│ │ ├── Article Title.md ← frontmatter contains summary / concepts / related
│ │ └── Article Title.html
│ ├── 05-AI Coding/
│ │ ├── Another Article.md
│ │ └── Another Article.html
│ └── 06-Unclassified/
├── Engineering_KnowBase/
├── Management_KnowBase/
├── PM_KnowBase/
├── Inbox/
│ └── daily_digest/ ← platform_collector daily summary (read and go, not into KB)
│ ├── 2026-05-06.md
│ └── 2026-05-07.md
└── Archive/
└── LocalDocs/
├── README.md ← Local document import process + frontmatter field query
├── .value_cache.json ← LLM value evaluation cache (by file_hash key)
└── imported/pdf/ ← All imported PDF original files
```
Classification and knowledge base names are completely customizable, see `common/kb_config.example.py`.
---
## Environmental Requirements
- Python 3.10+
- Playwright (for WeChat public account login status maintenance)
- yt-dlp (video subtitle extraction, used by video_collector)
- MarkItDown + pypdf (local PDF extraction, used by `tools/import_local_docs.py`, automatically installed by `run_import_local_docs.sh`)
- Claude Code CLI (optional, local PDF value evaluation defaults to it; if not installed, fallback to `ANTHROPIC_API_KEY` or add `--no-value-check`)
- DeepSeek API Key (optional, required for AI summary function of `platform_collector/digest_builder.py`; if not configured, skip summary and only output the original hotlist)
---
## About the Author
**Intelligent Code Exploration** — Continuously focus on AI engineering and efficiency tooling practices and thinking.
---
## License
MIT
