FNewsCrawler

# Tool List ## Due to unreasonable architecture design, maintenance has become extremely complex, and maintenance has been abandoned. Please switch to the new project: https://github.com/noimank/OmniData [](https://www.python.org/downloads/) [](https://fastapi.tiangolo.com/) [](https://modelcontextprotocol.io/) [](LICENSE) **Professional Financial Information MCP Service** - Providing real-time and accurate financial news data for large models. ## 🚀 Project Overview FNewsCrawler is a specialized financial information MCP (Model Context Protocol) service designed for large models. It uses efficient crawling technology to obtain real-time news from major financial websites, providing accurate and timely financial data support for AI models. ### 🎯 Core Features - **🤖 MCP Protocol Compatibility**: Fully compatible with the Model Context Protocol, seamlessly integrated into various AI applications. - **📈 Real-time Financial Data**: Supports obtaining the latest news from mainstream financial platforms such as Tonghuashun Wenda. - **🏗️ Enterprise-level Architecture**: Adopts a singleton pattern for browser management and context sharing mechanism. - **🔐 Intelligent Login System**: Supports QR code login and automatic maintenance of login status. - **⚡ High-performance Concurrency**: Based on asynchronous architecture, supports high concurrent access. - **🛡️ Production-level Stability**: Complete error handling, automatic retry, and health check mechanism. - **📊 Visualized Management**: Provides a web interface for tool management and system monitoring.  ## 🏛️ System Architecture ``` FNewsCrawler ├── 🧠 MCP Service Layer # Provides standardized interfaces for large models ├── 🕷️ Crawler Engine Layer # Multi-site financial data collection ├── 🌐 Browser Management Layer # Singleton browser instance management ├── 🔄 Context Sharing Layer # Domain-level session state management ├── 🔐 Login Authentication Layer # QR code login and state maintenance └── 💾 Data Storage Layer # Redis cache and state persistence ``` ## 📦 Quick Start ### Requirements - Python 3.12+ - Redis Server - Chrome/Chromium Browser ### Installation and Deployment #### 🐳 Docker Deployment (Recommended) **Method 1: Deploy from project source code** ```bash # Clone the project git clone https://github.com/noimank/FNewsCrawler.git cd FNewsCrawler # Build and start the service docker-compose up -d # View service status docker-compose ps # View logs docker-compose logs -f ``` **Method 2: Quick deployment and startup** ```bash # Pull the image from DockerHub and start in the background docker run --name fnewscrawler -p 8480:8480 -d noimankdocker/fnewscrawler:latest # View logs docker logs fnewscrawler -f ``` > 📚 **Detailed Documentation**: View [Docker Deployment Guide](docker/README.md) to understand complete deployment options, configuration instructions, and troubleshooting methods. #### 📦 Local Development and Testing 1. **Clone the project** ```bash git clone https://github.com/noimank/FNewsCrawler.git cd FNewsCrawler ``` 2. **Install dependencies** ```bash uv sync uv run playwright install chromium ``` 3. **Configure environment** ```bash # Create .env file, configure environment variables, and recommend disabling headless mode cp .env.example .env ``` 4. **Start Redis service** ```bash # Ubuntu/Debian sudo systemctl start redis-server # macOS brew services start redis # Windows # Download and start Redis service ``` 5. **Start the service** ```bash # Start the web service uv run python main.py ``` ### MCP Client Integration MCP access protocol supports sse and http, modified by environment variable MCP_SERVER_TYPE mcp service address：http://localhost:8480/mcp/mcp-server cherryStudio configuration： ``` { "mcpServers": { "fnewscrawler": { "url": "http://localhost:8480/mcp/mcp-server", "type": "streamableHttp" } } } ``` ### get method call Call endpoint：http://localhost:8480/api/mcp/call_tool/{tool_name}?xxxx Parameters： - tool_name：tool name - xxxx：tool parameters Return： - tool execution result, json format Example： ``` GET http://localhost:8480/api/mcp/call_tool/get_industry_stock_funds_flow_tool?industry_name=银行 ``` If the parameter type is list, use commas to separate Example： ``` GET http://localhost:8480/api/mcp/call_tool/news_crawl_batch?urls=http://example.com,http://example2.com ``` Additional supported functions: call any akshare function Call endpoint：http://localhost:8480/api/mcp/call_akshare/{fun_name}?xxxx Parameters： - fun_name：function name - xxxx：function parameters Supports processing return results： - duplicate_key：de-duplication field - drop_columns：drop fields, multiple fields separated by commas - return_type：return type, markdown or json, default markdown - filter_condition：filter condition string, similar to SQL syntax, implemented using pandas query statements, reference：https://gairuo.com/p/pandas-query - limit：return data row limit, default unlimited - sort_by：sort field, specify which column to sort by - ascending：sort order, true for ascending, false for descending, default true Return： - function execution result, json format, result format as follows: ```json { "success": true, "message": "Call tool stock_zh_a_gbjg_em successfully", "data": { "result": xxxxxfunction execution result, markdown format is string, json format is list[str]xxxx } } ``` Example： ``` GET http://localhost:8480/api/mcp/call_akshare/stock_zh_a_gbjg_em?symbol=603392.SH&return_type=json GET http://localhost:8480/api/mcp/call_akshare/stock_zh_a_gbjg_em?duplicate_key=变更日期&drop_columns=流通受限股份,变动原因&return_type=json&symbol=603392.SH GET http://localhost:8480/api/mcp/call_akshare/news_trade_notify_dividend_baidu?return_type=json&date=20240409&filter_condition=交易所 == "SZ" # Sorting and limiting examples GET http://localhost:8480/api/mcp/call_akshare/stock_zh_a_hist?symbol=000001&sort_by=日期&ascending=true&limit=10 GET http://localhost:8480/api/mcp/call_akshare/stock_zh_a_hist?symbol=000001&sort_by=收盘&ascending=false&limit=5&return_type=json GET http://localhost:8480/api/mcp/call_akshare/stock_zh_a_hist?symbol=000001&filter_condition=收盘 > 10&sort_by=成交量&ascending=false&limit=3&drop_columns=变动因素 ``` Error example： ``` GET http://localhost:8480/api/mcp/call_akshare/stock_zh_a_gbjg_em?symbol="603392.SH"&return_type=json ``` **Notes：** - Do not add quotes to parameters, neither single nor double quotes, except for filter_condition - sort_by field must be the actual column name in the returned data - limit must be a positive integer, ascending supports true/false, 1/0, yes/no and other formats > 📚 **Detailed Documentation**: View [GET API Interface Documentation](docs/get_api.md) to understand complete interface instructions, parameter configuration, and usage examples. ## 🛠️ Development Roadmap ### 🔄 Completed - [x] Tonghuashun Wenda news query interface - [x] MCP protocol standard support - [x] Tonghuashun industry fund flow query interface - [x] Tonghuashun concept fund flow query interface - [x] General news query interface - [x] Tonghuashun financial news, macroeconomic, industry news, international finance, financial market, company news, regional economy, financial comment, financial person query interface - [x] Tonghuashun Wenda A-share stock semantic screening tool - [x] Tonghuashun Wenda financing and securities information query tool - [x] Tonghuashun Wenda historical fund flow query tool - [x] Oriental Wealth certain industry historical fund flow query tool - [x] Oriental Wealth certain industry individual stock historical fund flow query tool - [x] Oriental Wealth large market fund flow query tool - [x] Oriental Wealth stock bulk transaction daily details query tool - [x] Oriental Wealth stock dragon tiger list query tool - [x] tushare data source support - [x] Stock technical indicator query interface - Moving average - Bollinger Bands - KDJ - MACD - RSI - VWMA - ATR - [x] Stock chip and win rate query interface (tushare) - [x] Stock chip distribution query interface (akshare) - [x] Access akshare data source - [x] Provide get endpoint to call any akshare function - [x] Northbound/Southbound fund flow ### 📈 Upcoming - [ ] Dragon tiger list data interface - Institutional transaction details - Business department ranking analysis - [ ] Market center interface - Real-time market push - Disk data analysis tips： Most interfaces if not made into MCP interfaces can be implemented by calling akshare functions using the get method. ### 🎯 Long-term Plan - [ ] Expand more financial data sources - [ ] Provide complete API documentation - [ ] Advanced data analysis tools The goal is to build a comprehensive financial information MCP service, providing real-time and accurate financial news data for large models. ## 🔧 Development Guide ### Add new financial data source 1. **Create crawler module** ```bash mkdir fnewscrawler/spiders/new_site_name touch fnewscrawler/spiders/new_site_name/__init__.py touch fnewscrawler/spiders/new_site_name/crawl.py touch fnewscrawler/spiders/new_site_name/login.py # If login is required ``` 2. **Implement login class** (if needed) ```python from fnewscrawler.core.qr_login_base import QRLoginBase class NewSiteLogin(QRLoginBase): async def get_qr_code(self, qr_type: str) -> Tuple[bool, str]: # Implement QR code acquisition logic pass async def verify_login_success(self) -> bool: # Implement login verification logic pass ``` 3. **Create MCP tool** ```python # fnewscrawler/mcp/new_site_name/crawl.py from fnewscrawler.mcp import mcp_server @mcp_server.tool() async def new_site_query(query: str): """New site query tool description""" # Implement query logic pass ``` 4. **Register tool** ```python # fnewscrawler/mcp/new_site_name/__init__.py import fnewscrawler.mcp.new_site_name.crawl ``` ### Core design principles - **Singleton pattern**: BrowserManager and ContextManager adopt the singleton pattern to ensure efficient resource utilization - **Domain isolation**: Each domain maintains an independent browser context to avoid session conflicts - **Asynchronous priority**: Fully adopt asynchronous programming to improve concurrent performance - **State persistence**: Login state and session information automatically saved to Redis - **Health check**: Regularly check browser and context health status, automatically recover exceptions ## 📊 System Monitoring Access `http://localhost:8480/monitor` to view system status: - 🌐 **Browser status**: Instance health, version information, resource usage - 🔄 **Context management**: Active sessions, idle time, usage statistics - 🛠️ **MCP tool**: Tool status, enable/disable management - 📈 **System logs**: Real-time log viewing and filtering ## 🔧 Tool Management Access `http://localhost:8480/mcp` for tool management: - ✅ **Enable/disable tools**: Dynamically control tool availability - 📝 **Tool details view**: View tool description and parameters - 🔄 **Batch operation**: Support batch enable/disable tools - 📊 **Usage statistics**: Tool call times and status monitoring ## 🤝 Contribution Guide We welcome community contributions! Please view [Development Documentation](docs/) for detailed information: - [Crawler development guide](docs/spider_development.md) - [MCP tool development guide](docs/mcp_tool_development.md) ## 📄 License This project adopts the MIT license - view [LICENSE](LICENSE) file for details. ## 🙏 Acknowledgments - [FastMCP](https://github.com/jlowin/fastmcp) - MCP protocol implementation - [Playwright](https://playwright.dev/) - Browser automation - [FastAPI](https://fastapi.tiangolo.com/) - Web framework - [Redis](https://redis.io/) - Data storage --- **Providing professional financial data support for large models** 🚀 If you have any questions or suggestions, please submit [Issue](https://github.com/noimank/FNewsCrawler/issues) or [Pull Request](https://github.com/noimank/FNewsCrawler/pulls).