Content
<div align="center">
# OpenHawk
### Global AI Intelligence Operating System for the Agent Era
From paper and model releases to capital markets, policy regulation, and open-source ecosystems,
uniformly crawl, deduplicate, analyze, subscribe, push, and provide programmable interfaces through MCP + CLI.
[](#)
[](#docker-startup)
[](#mcp-service)
[](#agent-cli-interface-newly-added)
[](LICENSE)
**📣 Multi-Channel Subscription Push**
**🏷️ Important Tags**
<p align="center">
<img src="image.png" alt="OpenHawk project banner" width="920" />
</p>
English | [简体中文](README-ZH.md) | [हिन्दी](README-HI.md) | [Español](README-ES.md) | [العربية](README-AR.md) | [Français](README-FR.md) | [Português](README-PT.md) | [বাংলা](README-BN.md) | [日本語](README-JA.md) | [한국어](README-KO.md)
</div>
## 🚀 Why We Built This Project
AI information is scattered across different sources and updated rapidly. Manual tracking usually encounters:
- Information fragmentation: papers, company announcements, financial reports, policies, and open-source trends are disjointed.
- Time-sensitive noise: old news reappears, making it difficult to filter out truly incremental information.
- Deduplication challenges: cross-source reposting leads to duplicate crawling and pushing.
- Insufficient automation: it's hard to directly integrate information retrieval into agent workflows.
`OpenHawk` aims to create a sustainable, scalable, and directly callable intelligence infrastructure.
## 👥 Who Is This For?
- AI researchers: continuously track paper and method evolution.
- Product/engineering teams: monitor model releases, open-source toolchains, and developer signals.
- Investment and commercial analysis teams: track AI-related financial reports, capital markets, and industry reports.
- Policy and compliance teams: monitor regulatory and security events across multiple regions.
- Agent builders: need programmable MCP/CLI tool interfaces.
## 🧭 Six Independent Pages (Parallel Relationship)
| Page | Primary Use | Typical Sources |
| --- | --- | --- |
| AI Paper Radar | Academic tracking and in-depth paper analysis | arXiv and other academic RSS feeds |
| AI Frontier Radar | Model/product/technology advancements | company websites, tech blogs, official news |
| AI Financial Information | Capital market and company dynamics | financial reports, earnings calls, market news, and investment announcements |
| AI Industry Report | Industry research insights | global institutional reports and whitepapers |
| AI Policy and Security | Regulatory and risk events | policy institutions, regulatory announcements, and security incident sources |
| AI Open Source Ecosystem and Developer Signal | Open-source tools and community signals | GitHub Trending and open-source project dynamics |
> These 6 pages are independent. Each page has its own parameters, subscriptions, and push configurations.
## ⚙️ Core Capabilities
- Multi-source crawling + regional organization (organizing channels by country/region and source type).
- Historical persistence and deduplication (avoiding duplicate crawling and pushing).
- Default freshness control (e.g., 90-day window) to reduce stale information noise.
- Unified translation pipeline (target language support), covering titles, LLM analysis, and push content.
- Multi-channel notifications: `feishu`, `wework`, `wechat`, `telegram`, `dingtalk`, `ntfy`, `bark`, `slack`, `email`.
- Intelligent push strategies: `daily`, `incremental`, `realtime`.
- Scheduled crawling and automatic subscription pushing.
- MCP tooling interface.
- Agent CLI (newly added) supports direct tool invocation.
## 🌐 Unified Translation Pipeline for Any Target Language
- Unified scope: same pipeline covers `title`, `LLM summary/analysis`, and `push body`, avoiding multiple translation logics.
- Target languages: supports `English`, `Korean`, `Japanese`, `French`, `Chinese`, `Traditional Chinese`, and custom languages.
- Cost control: batch translation, incremental completion, historical persistence reuse, and only translating missing or updated fields by default.
- Consistency: Web, MCP, and CLI use the same `output_language` configuration semantics, ensuring consistent language output in page display and notification content.
Example (CLI):
```bash
# Set the output language of the AI Finance page to Japanese
openhawk-ai-cli call save_scope_settings --args '{"scope":"market_finance","output_language":"Japanese"}'
# Trigger crawling and present content in the specified language
openhawk-ai-cli call fetch_scope_items --args '{"scope":"market_finance","max_per_source":20}'
```
## 🧠 Intelligent Push Strategies
| Strategy | Trigger Method | Applicable Scenarios | Characteristics |
| --- | --- | --- | --- |
| `daily` | Daily scheduled summary push | management daily reports, team morning reports | comprehensive topic aggregation, suitable for stable rhythms |
| `incremental` | Push new information within a timed window | daily monitoring | deduplicated, lower noise |
| `realtime` | Immediate push upon event trigger | major model releases, policy breaks, financing news | prioritized, high timeliness, no waiting |
Explanation:
- Strategies take effect independently based on "subscription rules" and can be configured separately on each of the 6 pages.
- Can be combined with filtering conditions (source, region, event type, keywords) for refined alerts.
## 🧱 System Architecture
```text
+---------------------------+
| Data Sources |
| papers / frontier / ... |
+-------------+-------------+
|
v
+----------------+ +------------+ +-------------------+
| Fetch & Dedupe |-->| Persistence|-->| Analysis & Routing|
+----------------+ +------------+ +-------------------+
| |
v v
+-------------+ +------------------+
| Web Console | | Notification Push |
+------+------+ +------------------+
|
+----------+----------+
| MCP Server / CLI |
| (agent automation) |
+---------------------+
```
## 🖼️ Project Screenshots
### Web Frontend Pages (img1 - img4)
| Page 1 | Page 2 |
| --- | --- |
|  |  |
|  |  |
### Channel Subscription Page (img5)
---
## ⚡ Quick Start
### 1) 🧩 Environment Requirements
- Python `>= 3.12`
- Recommended use of `uv`
- Docker users need to install `Docker` + `Docker Compose`
### 2) 🖥️ Local Startup (Development Mode)
```bash
uv sync --locked
```
1. Run the crawling main program (execute once):
```bash
openhawk-ai
```
2. Run the interactive Web console (6-page UI):
```bash
python -m openhawk_ai.web.panel_server --port 8080 --output-dir output
```
3. Start the MCP service (HTTP):
```bash
openhawk-ai-mcp --transport http --host 0.0.0.0 --port 3333
```
### 3) 🐳 Docker Startup
```bash
docker compose -f docker/docker-compose.yml up -d --build
```
Default ports:
- Main service (Web port mapping within the container): `WEBSERVER_PORT` (default 8080)
- MCP: `http://127.0.0.1:3333/mcp`
Stop:
```bash
docker compose -f docker/docker-compose.yml down
```
View logs:
```bash
docker compose -f docker/docker-compose.yml logs -f
```
---
## 🤖 Agent CLI Interface (Newly Added)
To enable direct tool invocation without MCP transport, `openhawk-ai-cli` has been added.
### 🎯 Design Goals
- Consistent with MCP tools (same tool names, parameter semantics).
- JSON input / JSON output for easy automation and orchestration.
- Suitable for Shell, CI, Python subprocess, and Agent executors.
### 🧪 Post-Installation Commands
```bash
openhawk-ai-cli tools
```
### 📌 Common Examples
1. List all callable tools and parameters:
```bash
openhawk-ai-cli tools
```
2. Invoke a tool with inline JSON parameters:
```bash
openhawk-ai-cli call list_scope_items --args '{"scope":"market_finance","limit":20}'
```
3. Invoke a tool with parameters from a file:
```bash
openhawk-ai-cli call upsert_scope_subscription --args-file ./payload.json
```
4. Specify the project root directory and output directory:
```bash
openhawk-ai-cli --project-root . --output-dir ./output call get_project_overview
```
5. Compact JSON output (suitable for pipeline processing):
```bash
openhawk-ai-cli call list_scopes --compact
```
### Windows PowerShell Example (Recommended)
1. Use `ConvertTo-Json` + `--args-file` (most reliable):
```powershell
$payload = @{ scope = "market_finance"; limit = 20 } | ConvertTo-Json -Compress
$payload | Set-Content -Encoding utf8 .\payload.json
openhawk-ai-cli call list_scope_items --args-file .\payload.json --compact
```
2. Write parameters using Here-String:
```powershell
@'
{
"scope": "frontier",
"max_per_source": 20,
"source_ids": ["openai-news", "anthropic-news"]
}
'@ | Set-Content -Encoding utf8 .\payload.json
openhawk-ai-cli call fetch_scope_items --args-file .\payload.json --compact
```
3. Tools without parameters can be called directly:
```powershell
openhawk-ai-cli call get_project_overview --compact
```
### 🧾 Exit Code Conventions
- `0`: successful invocation.
- `1`: execution exception.
- `2`: parameter error / tool not found / JSON format error.
### 🛠️ CLI Parameters and Coverage
The CLI has fixed parameters as follows:
| Level | Parameter | Description |
| --- | --- | --- |
| Global | `--project-root` | overrides project root directory |
| Global | `--output-dir` | overrides runtime output directory |
| Global | `--compact` | compact JSON output |
| Subcommand | `tools` | list callable tools |
| Subcommand | `call <tool>` | invoke a specified tool |
| `call` option | `--args` | inline JSON parameters |
| `call` option | `--args-file` | read parameters from a JSON file |
CLI business parameters are determined by specific tools. Execute the following command to view all tool parameter definitions:
```bash
openhawk-ai-cli tools --compact
```
Coverage:
- Covers all 22 tool capabilities exposed by MCP (project overview, page crawling, page settings, page subscriptions, paper analysis, etc.).
- Does not directly cover process/container lifecycle management (e.g., starting or stopping Docker, launching Web service processes).
- Does not directly cover frontend interaction behaviors (e.g., browser clicks), but covers corresponding data layer operations (settings, crawling, subscriptions, pushing).
---
## 🔌 MCP Service
### Startup
```bash
# stdio
python -m mcp_server.server --transport stdio
# http
python -m mcp_server.server --transport http --host 0.0.0.0 --port 3333
```
HTTP Endpoint:
`http://127.0.0.1:3333/mcp`
### MCP Tool Groups
1. Project Overview
- `get_project_overview`
- `list_pages`
- `list_scopes`
2. Global Settings
- `get_global_settings`
- `save_global_settings`
3. Page Data and Crawling
- `list_scope_sources`
- `list_scope_items`
- `fetch_scope_items`
- `get_scope_settings`
- `save_scope_settings`
4. Page Subscriptions
- `list_scope_subscriptions`
- `upsert_scope_subscription`
- `delete_scope_subscription`
- `run_scope_subscriptions`
5. Paper Capabilities
- `list_papers`
- `get_paper_detail`
- `deep_analyze_paper`
- `set_paper_action`
6. Paper Subscriptions
- `list_paper_subscriptions`
- `upsert_paper_subscription`
- `delete_paper_subscription`
- `run_paper_subscriptions`
---
## 🧠 Configuration Description
Main configuration directory: `config/`
Key files:
- `config/config.yaml`: Main configuration (crawling, pushing, AI, storage, etc.)
- `config/timeline.yaml`: Timing strategy (preset and custom)
- `config/frequency_words.txt`: Keyword rules
- `config/ai_interests.txt`: Interest direction definition
- `config/ai_analysis_prompt.txt`: Analysis prompt words
- `config/ai_translation_prompt.txt`: Translation prompt words
Runtime output directory: `output/`
Common persistent data:
- `output/ai_progress_items.json`
- `output/ai_progress_seen.json`
- `output/panel_settings.json`
- `output/progress_page_settings.json`
- `output/panel_subscriptions.json`
- `output/progress_subscriptions.json`
- `output/news/*.db`
- `output/rss/*.db`
---
## 📣 Push Channels
Already supported and unified in backend + frontend + MCP + CLI:
- `feishu`
- `wework`
- `wechat` (personal WeChat, based on WeCom text)
- `telegram`
- `dingtalk`
- `ntfy`
- `bark`
- `slack`
- `email`
---
## 🗂️ Project Structure
```text
.
├─ openhawk_ai/ # Core capabilities (crawling/analysis/notification/Web)
│ ├─ __main__.py # Main program entry
│ ├─ agent_cli.py # Agent CLI entry (new)
│ └─ web/panel_server.py # Interactive console service
├─ mcp_server/ # MCP server
├─ config/ # Configuration and prompts
├─ docker/ # Dockerfile / compose / entry script
├─ docs/assets/ # README visual resources (including OpenHawk SVG)
├─ output/ # Runtime persistent data
├─ README.md
└─ README-EN.md
```
---
## ❓Frequently Asked Questions
### Q1: What is the relationship between CLI and MCP?
CLI directly calls MCP isomorphic tool functions; MCP is suitable for protocol access, and CLI is suitable for scripting and local automation.
### Q2: Can CLI cover all features?
CLI can cover all currently exposed MCP capabilities (22 tools). If you can do data layer operations on the UI (crawling, checking, setting, subscribing, pushing), you can usually implement them through CLI.
### Q3: Why do I see duplicate messages?
Please check:
- Whether `output/ai_progress_seen.json` is mounted and persisted.
- Whether multiple duplicate subscription rules are enabled.
- Whether `timeline` and page-level subscriptions trigger repeatedly in the same time window.
### Q4: How to minimize Agent access?
The simplest way: Let Agent call
```bash
openhawk-ai-cli call get_project_overview
```
Then call `list_scope_items` / `list_papers` / `run_*_subscriptions` as needed.
---
## 🙏 Acknowledgements and References
- This project references and acknowledges [TrendRadar](https://github.com/sansan0/TrendRadar) in product ideas and engineering practices.
- OpenHawk has independently evolved on this basis, including 6-page parallel system, regional data sources, multi-channel subscription push, MCP + Agent CLI integration, etc.
---
## 📄 License
This project uses the [MIT License](LICENSE).
