Content
<p align="center">
<h1 align="center">🔬 AutoResearch MCP</h1>
<p align="center">
<strong>Frozen Evaluation 기반 자율 반복 개선 에이전트 시스템</strong>
</p>
<p align="center">
<a href="#quickstart">Quickstart</a> · <a href="#architecture">Architecture</a> · <a href="#how-it-works">How it Works</a> · <a href="#cli-reference">CLI Reference</a>
</p>
</p>
<p align="center">
<img src="https://img.shields.io/badge/python-3.11+-blue?logo=python&logoColor=white" alt="Python 3.11+">
<img src="https://img.shields.io/badge/tests-81%20passed-brightgreen?logo=pytest" alt="Tests">
<img src="https://img.shields.io/badge/E2E-34%20passed-brightgreen" alt="E2E">
<img src="https://img.shields.io/badge/Claude%20CLI-OAuth-blueviolet?logo=anthropic" alt="Claude CLI">
<img src="https://img.shields.io/badge/license-MIT-green" alt="MIT License">
</p>
---
## 📌 What is this?
When a goal is defined, this system enables **the LLM to autonomously modify the code**, measure it with frozen evaluation, and maintain or rollback changes based on performance.
```
┌─────────────────────────────────────────────────────┐
│ 🎯 Goal: "Improve search relevance" │
│ │
│ Iteration 1: Select H-002 → Claude modifies code │
│ → Tests pass → score 0.5→1.0 │
│ → ✅ ACCEPT → baseline updated │
│ │
│ Iteration 2: Select H-003 → Claude modifies code │
│ → Tests fail → ❌ REJECT → rollback │
│ │
│ ... Repeat until goal achievement or hypothesis exhaustion │
└─────────────────────────────────────────────────────┘
```
> **No API key required** — Works with Claude Code CLI or OpenAI Codex CLI using OAuth login.
---
## ✨ Key Features
| Feature | Description |
|---------|-------------|
| 🤖 **LLM Code Modification** | Claude Code CLI actually modifies files (`--dangerously-skip-permissions`) |
| 🧊 **Frozen Eval** | Evaluation criteria remain unchanged during iterations — ensuring fair measurement |
| 🔄 **Automatic Accept/Reject** | 10-step Decision Engine evaluates score, tests, and constraint violations |
| ⏪ **Automatic Rollback** | Reverts changes with `git restore` upon rejection (agent state preserved) |
| 🧬 **Hypothesis Lifecycle** | Manages hypothesis states: selected → tried → accepted/rejected |
| 📊 **Complete Observability** | RESULTS.tsv, DECISIONS.md, MEMORY.md, ITERATION_STATE.json |
| 🔒 **Safety Checks** | Forbidden path, scope violation, and change budget enforcement |
| 🏗️ **Constraint Pipeline** | Latency budget, regression detection, and score sanity checks |
---
## 🚀 Quickstart
### 1. Prerequisites
```bash
# Python 3.11+
uv venv .venv --python 3.12
uv pip install pytest
# Claude Code CLI (OAuth — no API key required)
npm install -g @anthropic-ai/claude-code
claude auth login # Login with Claude account in browser
# Alternatively, OpenAI Codex CLI
npm install -g @openai/codex
codex login # Login with ChatGPT account in browser
```
### 2. Authentication Check
```bash
python orchestrator/cli.py --provider claude auth
# [claude] OK: Claude OAuth credentials found
python orchestrator/cli.py --provider codex auth
# [codex] OK: Codex OAuth credentials found
```
### 3. Run
```bash
# Single iteration
python orchestrator/cli.py single --iteration 1 --baseline 0.5
# Loop (max 10 iterations, early termination on stagnation)
python orchestrator/cli.py --allow-dirty loop --max-iterations 10
# Terminate when target score reached
python orchestrator/cli.py --allow-dirty loop --max-iterations 50 --target-score 0.95
# Use Codex CLI
python orchestrator/cli.py --provider codex --allow-dirty loop --max-iterations 5
```
---
## 🏗️ Architecture
```
autoresearch-mcp/
├── 🧠 orchestrator/ # Python orchestrator (main execution path)
│ ├── cli.py # CLI entry point (single / loop / auth)
│ ├── runner.py # 12-Phase IterationRunner
│ ├── loop.py # LoopOrchestrator (stagnation, target)
│ ├── agents.py # 7 agent functions + Claude/Codex CLI integration
│ ├── state.py # 16 Phase, 11 DecisionCode, IterationState
│ ├── config.py # OrchestratorConfig (provider, paths, limits)
│ └── logging.py # RESULTS.tsv, DECISIONS.md, MEMORY.md
│
├── 📦 src/ # Product code (target for LLM modifications)
│ └── query_processor.py # normalize_query() — target for improvement
│
├── 🧊 eval/ # Frozen Evaluation (immutable)
│ ├── frozen_eval.py # Fixed evaluation script
│ ├── fixtures.json # Fixed test inputs (3 queries)
│ ├── baseline.json # Current baseline score
│ ├── constraints.py # Latency/regression constraint checks
│ └── rubric.md # Score criteria explanation
│
├── 📋 agent/ # Agent state files
│ ├── PRODUCT_GOAL.md # Top-level goal
│ ├── TASK.md # Current cycle goal
│ ├── RULES.md # Operational rules
│ ├── HYPOTHESES.md # Hypothesis registry (lifecycle management)
│ ├── PLAN.md # Current execution plan
│ ├── MEMORY.md # Accumulated learnings (accepted/rejected patterns)
│ ├── ITERATION_STATE.json # Current phase state
│ ├── RESULTS.tsv # All iteration result logs
│ └── DECISIONS.md # Decision history
│
├── 📝 prompts/ # Agent prompts
│ ├── implementer.md # ✅ Passed to Claude CLI
│ ├── explorer.md # 📌 Future LLM integration
│ ├── planner.md # 📌 Future LLM integration
│ ├── critic.md # 📌 Future LLM integration
│ ├── controller.md # Sufficient with rule-based approach
│ └── archivist.md # Sufficient with rule-based approach
│
├── 🧪 tests/ # Tests (81)
│ ├── test_frozen_eval.py # Frozen eval verification
│ ├── test_orchestrator.py # Orchestrator overall verification
│ └── test_query_processor.py # Product code edge cases
│
├── 🔧 scripts/ # Shell scripts (non-standard, compatibility)
├── 📊 demo_test/ # E2E verification scripts (34 checks)
├── 📖 docs/ # Design documents (11, 307KB)
└── ⚙️ mcp/ # MCP server settings
```
---
## ⚙️ How it Works
### 12-Phase Iteration Pipeline
```
INIT → READ_CONTEXT → EXPLORE → PLAN → IMPLEMENT → RUN_TESTS
→ RUN_EVAL → CRITIQUE → DECIDE → ACCEPT/REJECT → ARCHIVE → DONE
```
| Phase | Agent | Action |
|-------|-------|--------|
| `INIT` | — | Initialize IterationState |
| `READ_CONTEXT` | — | Load agent/*.md files |
| `EXPLORE` | Explorer | Select actionable hypothesis from HYPOTHESES.md |
| `PLAN` | Planner | Parse change scope/tests from PLAN.md |
| `IMPLEMENT` | Implementer | **Modify code with Claude CLI** |
| `RUN_TESTS` | — | Execute `pytest tests/ -v` |
| `RUN_EVAL` | — | Execute `frozen_eval.py` + `constraints.py` |
| `CRITIQUE` | Critic | Review based on 8 rules (narrow win, scope, latency, etc.) |
| `DECIDE` | Controller | **10-step Decision Engine** |
| `ACCEPT/REJECT` | — | Update baseline or rollback changes |
| `ARCHIVE` | Archivist | Record RESULTS.tsv, DECISIONS.md, MEMORY.md |
| `DONE` | — | Update hypothesis state (accepted/rejected/tried) |
### 10-Level Decision Engine
```
Level 0: NO_CODE_CHANGE → No actual file changes (placeholder)
Level 1: TEST_FAIL → Test failure
Level 2: CONSTRAINT_FAIL → Latency/regression constraint violation
Level 3: FORBIDDEN_FILE → Attempt to modify eval/frozen_eval.py, etc.
Level 4: SCOPE_VIOLATION → File modification outside planned scope
Level 5: CRITIC_BLOCK → Critic severity=high
Level 6: SCORE_REGRESSION → Score degradation
Level 7: NO_IMPROVEMENT → Score remains the same
Level 8: ACCEPT → All checks passed + score improvement
```
### Hypothesis Lifecycle
```
proposed → selected → (iteration) → accepted ✅
→ rejected ❌
→ tried (NO_CODE_CHANGE 2 consecutive)
parked → (manual activation required)
```
---
## 📋 CLI Reference
```bash
python orchestrator/cli.py [OPTIONS] COMMAND [ARGS]
```
### Global Options
| Option | Values | Default | Description |
|--------|--------|---------|-------------|
| `--provider` | `claude`, `codex` | `claude` | LLM provider (OAuth-based) |
| `--mode` | `single-agent` | `single-agent` | Execution mode |
| `--allow-dirty` | flag | `false` | Allow git dirty state |
### Commands
| Command | Description | Example |
|---------|-------------|---------|
| `single` | Run single iteration | `cli.py single --iteration 1 --baseline 0.5` |
| `loop` | Run loop | `cli.py loop --max-iterations 10 --target-score 0.9` |
| `auth` | Verify LLM authentication | `cli.py --provider claude auth` |
### Single Options
| Option | Type | Default | Description |
|--------|------|---------|-------------|
| `--iteration` | int | `1` | Iteration number |
| `--baseline` | float | `0.0` | Baseline score |
| `--hypothesis` | str | `H-001` | Starting hypothesis |
### Loop Options
| Option | Type | Default | Description |
|--------|------|---------|-------------|
| `--max-iterations` | int | `10` | Maximum iterations |
| `--target-score` | float | — | Target score (terminate when reached) |
---
## 🔐 LLM Authentication
No API key required. Works with OAuth login.
### Claude Code CLI
```bash
# Install
npm install -g @anthropic-ai/claude-code
# OAuth login (Claude account in browser)
claude auth login
# Verify authentication
python orchestrator/cli.py --provider claude auth
```
> Token stored in `~/.claude/.credentials.json` (1-year validity).
### OpenAI Codex CLI
```bash
# Install
npm install -g @openai/codex
# OAuth login (ChatGPT account in browser)
codex login
# Verify authentication
python orchestrator/cli.py --provider codex auth
```
> Token stored in `~/.codex/auth.json` (auto-renewal).
---
## 🧪 Testing
```bash
# Unit tests (81)
uv run pytest tests/ -v
# E2E pipeline test (34 checks)
uv run python demo_test/run_e2e_test.py
# Actual Claude CLI accept path verification (OAuth required)
# → src/query_processor.py modified by Claude → score 0.5→1.0 → ACCEPT
python orchestrator/cli.py --allow-dirty single --iteration 1 --baseline 0.5
```
### Test Coverage
| Test Suite | Count | Covers |
|-----------|-------|--------|
| `test_frozen_eval.py` | 9 | frozen eval scoring, fixture correctness |
| `test_query_processor.py` | 13 | normalize_query edge cases |
| `test_orchestrator.py` | 52 | state, decision engine, critic, explorer, planner, logging, runner, loop |
| `demo_test/run_e2e_test.py` | 34 | full pipeline (reject + stagnation + constraints) |
---
## 📊 Observability
### Iteration Results
```bash
# Overall result log (TSV, 15 columns)
cat agent/RESULTS.tsv
# Decision history
cat agent/DECISIONS.md
# Current state
cat agent/ITERATION_STATE.json
# Cumulative learning
cat agent/MEMORY.md
# Hypothesis state
cat agent/HYPOTHESES.md
# Generate final report
python scripts/make_final_report.py
```
### ITERATION_STATE.json Example
```json
{
"iteration": 1,
"phase": "done",
"selected_hypothesis": "H-002",
"tests_pass": true,
"candidate_score": 1.0,
"decision": "accept"
}
```
---
## 📐 Rules & Safety
### Absolute Rules
| Rule | Enforced by |
|------|-------------|
| 🚫 `eval/frozen_eval.py` modification prohibited | FORBIDDEN_FILE (Level 3) |
| 🚫 `eval/fixtures.json` modification prohibited | FORBIDDEN_FILE (Level 3) |
| 🚫 Direct modification of `eval/baseline.json` prohibited | FORBIDDEN_FILE (Level 3) |
| 📏 1 iteration = 1 core change | Critic + change budget |
| ⏱️ Latency increase ≤ 5% | CONSTRAINT_FAIL (Level 2) |
### Termination Conditions
| Condition | Default |
|-----------|---------|
| Target score reached | `--target-score` |
| N consecutive non-improvements | 3 |
| Maximum iterations reached | `--max-iterations` |
| All hypotheses exhausted | Automatic detection |
---
## 🗺️ Roadmap
- [x] Single-agent pipeline
- [x] Claude Code CLI integration (OAuth)
- [x] Codex CLI support
- [x] 10-level Decision Engine
- [x] Hypothesis lifecycle management
- [x] Constraints pipeline (latency, regressions)
- [x] Windows compatibility (cp949 fix)
- [ ] Explorer LLM connection (dynamic hypothesis generation)
- [ ] Planner LLM connection (dynamic planning per hypothesis)
- [ ] Critic LLM connection (LLM-based regression analysis)
- [ ] Multi-agent parallel execution
---
## 📄 License
MIT License
---
<p align="center">
Built with 🔬 <a href="https://github.com/coreline-ai/autoresearch-mcp-codex">AutoResearch MCP</a>
</p>
