Content
# Tool List
## Due to unreasonable architecture design, maintenance has become extremely complex, and maintenance has been abandoned. Please switch to the new project: https://github.com/noimank/OmniData
[](https://www.python.org/downloads/)
[](https://fastapi.tiangolo.com/)
[](https://modelcontextprotocol.io/)
[](LICENSE)
**Professional Financial Information MCP Service** - Providing real-time and accurate financial news data for large models.
## 🚀 Project Overview
FNewsCrawler is a specialized financial information MCP (Model Context Protocol) service designed for large models. It uses efficient crawling technology to obtain real-time news from major financial websites, providing accurate and timely financial data support for AI models.
### 🎯 Core Features
- **🤖 MCP Protocol Compatibility**: Fully compatible with the Model Context Protocol, seamlessly integrated into various AI applications.
- **📈 Real-time Financial Data**: Supports obtaining the latest news from mainstream financial platforms such as Tonghuashun Wenda.
- **🏗️ Enterprise-level Architecture**: Adopts a singleton pattern for browser management and context sharing mechanism.
- **🔐 Intelligent Login System**: Supports QR code login and automatic maintenance of login status.
- **⚡ High-performance Concurrency**: Based on asynchronous architecture, supports high concurrent access.
- **🛡️ Production-level Stability**: Complete error handling, automatic retry, and health check mechanism.
- **📊 Visualized Management**: Provides a web interface for tool management and system monitoring.

## 🏛️ System Architecture
```
FNewsCrawler
├── 🧠 MCP Service Layer # Provides standardized interfaces for large models
├── 🕷️ Crawler Engine Layer # Multi-site financial data collection
├── 🌐 Browser Management Layer # Singleton browser instance management
├── 🔄 Context Sharing Layer # Domain-level session state management
├── 🔐 Login Authentication Layer # QR code login and state maintenance
└── 💾 Data Storage Layer # Redis cache and state persistence
```
## 📦 Quick Start
### Requirements
- Python 3.12+
- Redis Server
- Chrome/Chromium Browser
### Installation and Deployment
#### 🐳 Docker Deployment (Recommended)
**Method 1: Deploy from project source code**
```bash
# Clone the project
git clone https://github.com/noimank/FNewsCrawler.git
cd FNewsCrawler
# Build and start the service
docker-compose up -d
# View service status
docker-compose ps
# View logs
docker-compose logs -f
```
**Method 2: Quick deployment and startup**
```bash
# Pull the image from DockerHub and start in the background
docker run --name fnewscrawler -p 8480:8480 -d noimankdocker/fnewscrawler:latest
# View logs
docker logs fnewscrawler -f
```
> 📚 **Detailed Documentation**: View [Docker Deployment Guide](docker/README.md) to understand complete deployment options, configuration instructions, and troubleshooting methods.
#### 📦 Local Development and Testing
1. **Clone the project**
```bash
git clone https://github.com/noimank/FNewsCrawler.git
cd FNewsCrawler
```
2. **Install dependencies**
```bash
uv sync
uv run playwright install chromium
```
3. **Configure environment**
```bash
# Create .env file, configure environment variables, and recommend disabling headless mode
cp .env.example .env
```
4. **Start Redis service**
```bash
# Ubuntu/Debian
sudo systemctl start redis-server
# macOS
brew services start redis
# Windows
# Download and start Redis service
```
5. **Start the service**
```bash
# Start the web service
uv run python main.py
```
### MCP Client Integration
MCP access protocol supports sse and http, modified by environment variable MCP_SERVER_TYPE
mcp service address:http://localhost:8480/mcp/mcp-server
cherryStudio configuration:
```
{
"mcpServers": {
"fnewscrawler": {
"url": "http://localhost:8480/mcp/mcp-server",
"type": "streamableHttp"
}
}
}
```
### get method call
Call endpoint:http://localhost:8480/api/mcp/call_tool/{tool_name}?xxxx
Parameters:
- tool_name:tool name
- xxxx:tool parameters
Return:
- tool execution result, json format
Example:
```
GET http://localhost:8480/api/mcp/call_tool/get_industry_stock_funds_flow_tool?industry_name=银行
```
If the parameter type is list, use commas to separate
Example:
```
GET http://localhost:8480/api/mcp/call_tool/news_crawl_batch?urls=http://example.com,http://example2.com
```
Additional supported functions: call any akshare function
Call endpoint:http://localhost:8480/api/mcp/call_akshare/{fun_name}?xxxx
Parameters:
- fun_name:function name
- xxxx:function parameters
Supports processing return results:
- duplicate_key:de-duplication field
- drop_columns:drop fields, multiple fields separated by commas
- return_type:return type, markdown or json, default markdown
- filter_condition:filter condition string, similar to SQL syntax, implemented using pandas query statements, reference:https://gairuo.com/p/pandas-query
- limit:return data row limit, default unlimited
- sort_by:sort field, specify which column to sort by
- ascending:sort order, true for ascending, false for descending, default true
Return:
- function execution result, json format, result format as follows:
```json
{
"success": true,
"message": "Call tool stock_zh_a_gbjg_em successfully",
"data": {
"result": xxxxxfunction execution result, markdown format is string, json format is list[str]xxxx
}
}
```
Example:
```
GET http://localhost:8480/api/mcp/call_akshare/stock_zh_a_gbjg_em?symbol=603392.SH&return_type=json
GET http://localhost:8480/api/mcp/call_akshare/stock_zh_a_gbjg_em?duplicate_key=变更日期&drop_columns=流通受限股份,变动原因&return_type=json&symbol=603392.SH
GET http://localhost:8480/api/mcp/call_akshare/news_trade_notify_dividend_baidu?return_type=json&date=20240409&filter_condition=交易所 == "SZ"
# Sorting and limiting examples
GET http://localhost:8480/api/mcp/call_akshare/stock_zh_a_hist?symbol=000001&sort_by=日期&ascending=true&limit=10
GET http://localhost:8480/api/mcp/call_akshare/stock_zh_a_hist?symbol=000001&sort_by=收盘&ascending=false&limit=5&return_type=json
GET http://localhost:8480/api/mcp/call_akshare/stock_zh_a_hist?symbol=000001&filter_condition=收盘 > 10&sort_by=成交量&ascending=false&limit=3&drop_columns=变动因素
```
Error example:
```
GET http://localhost:8480/api/mcp/call_akshare/stock_zh_a_gbjg_em?symbol="603392.SH"&return_type=json
```
**Notes:**
- Do not add quotes to parameters, neither single nor double quotes, except for filter_condition
- sort_by field must be the actual column name in the returned data
- limit must be a positive integer, ascending supports true/false, 1/0, yes/no and other formats
> 📚 **Detailed Documentation**: View [GET API Interface Documentation](docs/get_api.md) to understand complete interface instructions, parameter configuration, and usage examples.
## 🛠️ Development Roadmap
### 🔄 Completed
- [x] Tonghuashun Wenda news query interface
- [x] MCP protocol standard support
- [x] Tonghuashun industry fund flow query interface
- [x] Tonghuashun concept fund flow query interface
- [x] General news query interface
- [x] Tonghuashun financial news, macroeconomic, industry news, international finance, financial market, company news, regional economy, financial comment, financial person query interface
- [x] Tonghuashun Wenda A-share stock semantic screening tool
- [x] Tonghuashun Wenda financing and securities information query tool
- [x] Tonghuashun Wenda historical fund flow query tool
- [x] Oriental Wealth certain industry historical fund flow query tool
- [x] Oriental Wealth certain industry individual stock historical fund flow query tool
- [x] Oriental Wealth large market fund flow query tool
- [x] Oriental Wealth stock bulk transaction daily details query tool
- [x] Oriental Wealth stock dragon tiger list query tool
- [x] tushare data source support
- [x] Stock technical indicator query interface
- Moving average
- Bollinger Bands
- KDJ
- MACD
- RSI
- VWMA
- ATR
- [x] Stock chip and win rate query interface (tushare)
- [x] Stock chip distribution query interface (akshare)
- [x] Access akshare data source
- [x] Provide get endpoint to call any akshare function
- [x] Northbound/Southbound fund flow
### 📈 Upcoming
- [ ] Dragon tiger list data interface
- Institutional transaction details
- Business department ranking analysis
- [ ] Market center interface
- Real-time market push
- Disk data analysis
tips:
Most interfaces if not made into MCP interfaces can be implemented by calling akshare functions using the get method.
### 🎯 Long-term Plan
- [ ] Expand more financial data sources
- [ ] Provide complete API documentation
- [ ] Advanced data analysis tools
The goal is to build a comprehensive financial information MCP service, providing real-time and accurate financial news data for large models.
## 🔧 Development Guide
### Add new financial data source
1. **Create crawler module**
```bash
mkdir fnewscrawler/spiders/new_site_name
touch fnewscrawler/spiders/new_site_name/__init__.py
touch fnewscrawler/spiders/new_site_name/crawl.py
touch fnewscrawler/spiders/new_site_name/login.py # If login is required
```
2. **Implement login class** (if needed)
```python
from fnewscrawler.core.qr_login_base import QRLoginBase
class NewSiteLogin(QRLoginBase):
async def get_qr_code(self, qr_type: str) -> Tuple[bool, str]:
# Implement QR code acquisition logic
pass
async def verify_login_success(self) -> bool:
# Implement login verification logic
pass
```
3. **Create MCP tool**
```python
# fnewscrawler/mcp/new_site_name/crawl.py
from fnewscrawler.mcp import mcp_server
@mcp_server.tool()
async def new_site_query(query: str):
"""New site query tool description"""
# Implement query logic
pass
```
4. **Register tool**
```python
# fnewscrawler/mcp/new_site_name/__init__.py
import fnewscrawler.mcp.new_site_name.crawl
```
### Core design principles
- **Singleton pattern**: BrowserManager and ContextManager adopt the singleton pattern to ensure efficient resource utilization
- **Domain isolation**: Each domain maintains an independent browser context to avoid session conflicts
- **Asynchronous priority**: Fully adopt asynchronous programming to improve concurrent performance
- **State persistence**: Login state and session information automatically saved to Redis
- **Health check**: Regularly check browser and context health status, automatically recover exceptions
## 📊 System Monitoring
Access `http://localhost:8480/monitor` to view system status:
- 🌐 **Browser status**: Instance health, version information, resource usage
- 🔄 **Context management**: Active sessions, idle time, usage statistics
- 🛠️ **MCP tool**: Tool status, enable/disable management
- 📈 **System logs**: Real-time log viewing and filtering
## 🔧 Tool Management
Access `http://localhost:8480/mcp` for tool management:
- ✅ **Enable/disable tools**: Dynamically control tool availability
- 📝 **Tool details view**: View tool description and parameters
- 🔄 **Batch operation**: Support batch enable/disable tools
- 📊 **Usage statistics**: Tool call times and status monitoring
## 🤝 Contribution Guide
We welcome community contributions! Please view [Development Documentation](docs/) for detailed information:
- [Crawler development guide](docs/spider_development.md)
- [MCP tool development guide](docs/mcp_tool_development.md)
## 📄 License
This project adopts the MIT license - view [LICENSE](LICENSE) file for details.
## 🙏 Acknowledgments
- [FastMCP](https://github.com/jlowin/fastmcp) - MCP protocol implementation
- [Playwright](https://playwright.dev/) - Browser automation
- [FastAPI](https://fastapi.tiangolo.com/) - Web framework
- [Redis](https://redis.io/) - Data storage
---
**Providing professional financial data support for large models** 🚀
If you have any questions or suggestions, please submit [Issue](https://github.com/noimank/FNewsCrawler/issues) or [Pull Request](https://github.com/noimank/FNewsCrawler/pulls).
Connection Info
You Might Also Like
Vibe-Trading
Vibe-Trading: Your Personal Trading Agent
valuecell
Valuecell is a Python project for efficient data management.
hexstrike-ai
HexStrike AI is an AI-powered MCP cybersecurity automation platform with 150+ tools.
tradingview-mcp
AI-assisted TradingView chart analysis — connect Claude Code to your...
tradingview-mcp
TradingView MCP Server offers real-time market analysis for crypto and stocks.
AP2
AP2 provides code samples and demos for the Agent Payments Protocol.