Morpheus.AI

# Morpheus.AI This repository accompanies the Morpheus.AI benchmark study and also serves as a general-purpose Morpheus modeling toolkit. The current workflow uses the Codex SDK, a reusable Morpheus skill, and a thin Python MCP server for deterministic tasks such as PDF staging, XML writing, Morpheus execution, and output inspection. At the repository level there are three main pieces: - a benchmark runner that tests whether an agent can recreate Morpheus models from published papers; - a reusable Morpheus skill and MCP tool layer that can also be used for broader modeling tasks beyond benchmarking; - an analysis script that converts completed benchmark folders into CSV summaries and publication-oriented figures. ## Installation The repository does not include a `.venv/` directory. That environment is created locally after cloning. Minimal setup from the repository root: ```powershell python -m venv .venv .venv\Scripts\Activate.ps1 pip install -r requirements.txt npm install ``` The Python environment includes the MCP/PDF utilities plus common postprocessing packages (`numpy`, `pandas`, `matplotlib`, `scipy`, and `Pillow`) so benchmark agents can analyze generated CSVs/images and render final plots inside the run directory. If PowerShell blocks the activation script, run: ```powershell Set-ExecutionPolicy -Scope Process -ExecutionPolicy Bypass .venv\Scripts\Activate.ps1 ``` For shell-specific activation instructions and the full benchmark setup, see [SETUP.md](SETUP.md). ## Quick start 1. Create and activate `.venv`, then install Python and Node dependencies. 2. Confirm that `codex login status` and `morpheus --help` both work. 3. Run `npm run benchmark -- --max-papers 1`. 4. Run `python analyze_benchmark.py benchmark_runs/<timestamp>_benchmark`. 5. Inspect the new timestamped benchmark folder under `benchmark_runs/`. ## What the benchmark does For each PDF in `benchmark_papers/`, the runner can run one or more benchmark capability modes. The default mode is `morpheus_mcp`, which preserves the original full harness behavior. The available modes are: - `python_web`: web search + Python only. The agent must still produce MorpheusML `model.xml`; the host runs Morpheus afterward for technical scoring. - `morpheus_cli`: web search + Python + Morpheus CLI, without the Morpheus skill or MCP tools. - `morpheus_cli_docs`: Python + Morpheus CLI with internet disabled, using only staged MorpheusML documentation and the basic model template. - `morpheus_skill`: web search + Python + Morpheus CLI + Morpheus skill, without MCP tools. - `morpheus_mcp`: web search + Python + Morpheus CLI + Morpheus skill + Morpheus MCP tools. For each selected paper, mode, and repetition, the runner: 1. creates a timestamped benchmark directory and one isolated paper/mode/repetition run directory inside it; 2. stages the PDF, extracted paper text, focus prompt, and a visual manifest in an isolated `agent_workspace`; 3. detects likely figure and table pages but does not attach all page images up front; 4. exposes only the capabilities allowed by the selected benchmark mode; 5. lets the agent write MorpheusML, create fresh attempt folders, inspect stdout/stderr, CSVs, plots, and local PNGs directly, and revise if needed; 6. records attempt-level self-scores, model XML versions, technical evaluation files, and reproduction-oriented evaluation files; 7. runs one automatic recovery pass with `--continue-incomplete` after normal runs, then regenerates analysis tables and figures unless disabled. Two score tracks are written for each paper: - `technical_evaluation.json`: the legacy 0-7 executability score. - `reproduction_report.json`: a structured 0-8 reproduction assessment with four 0-2 criteria: - `source_coverage` - `mechanism_mapping` - `observable_alignment` - `parameter_plausibility` The default runner concurrency is one job at a time, but paper/mode/repetition jobs are independent and can be run in parallel with `--max-concurrent-jobs`. Each benchmark invocation writes benchmark-level diagnostics in the timestamped benchmark directory. It does not modify the source PDFs or the reference corpus. ## Repository layout - `benchmark_papers/` Benchmark inputs. The runner scans this folder for PDF files. - `benchmark_focus/` Optional per-paper focus prompts. A file named like the PDF stem, for example `10_Berkhout2025.txt`, is appended to that paper's benchmark prompt. - `references/` Morpheus examples and raw reference assets. These help the agent understand Morpheus, but they are not benchmark targets. - `.agents/skills/morpheus/` The publishable Morpheus skill used by the benchmark and intended for general Morpheus modeling tasks beyond paper recreation. - `src/benchmark/` TypeScript benchmark runner built on the Codex SDK. - `server.py` MCP server entry point. The supported transport is stdio. - `morpheus_mcp_server.py` Python implementation of the deterministic Morpheus and PDF utilities. - `morpheus_tools_cli.py` JSON bridge used by the benchmark runner. - `.mcp.json` Machine-readable MCP startup contract for local clients. - `benchmark.config.json` Default benchmark configuration. - `Archive/` Legacy Anthropic-era prompts, runners, servers, historical outputs, and superseded benchmark input corpora retained for reference only. ## Minimal benchmark run Start with one paper instead of the full benchmark: ```powershell npm run benchmark -- --max-papers 1 ``` If that succeeds, you should see: - a new run folder under `benchmark_runs/`; - `benchmark_summary.json` and `benchmark_manifest.json` in that run folder; - a `model.xml` generated for that paper; - Morpheus attempt logs under host `attempts/attempt_###/` and, for agent-run attempts, under `agent_workspace/.../attempt_###/`; - `technical_evaluation.json`; - `reproduction_report.json`; - transcript files under `transcripts/`; - a paper-specific subfolder containing the generated model, reports, logs, and transcripts. To run the full benchmark set: ```powershell npm run benchmark ``` To run a specific capability mode: ```powershell npm run benchmark -- --benchmark-mode python_web ``` To run all capability modes as a comparison matrix: ```powershell npm run benchmark -- --all-benchmark-modes ``` To see available CLI options: ```powershell npm run benchmark -- --help ``` ## Benchmark CLI reference The benchmark entry point is: ```powershell npm run benchmark -- [options] ``` Core run selection: - `--config <path>`: load a non-default benchmark config file. - `--benchmark-mode <python_web|morpheus_cli|morpheus_cli_docs|morpheus_skill|morpheus_mcp>`: select one mode; repeat the flag to select multiple modes. - `--all-benchmark-modes`: run all supported modes. - `--papers-dir <path>`: override the PDF input folder. - `--results-dir <path>`: override the parent output folder. - `--benchmark-focus-dir <path>`: override the focus prompt folder. - `--no-benchmark-focus`: disable focus prompts. - `--max-papers <number>`: run only the first N papers after sorting. - `--start-at-paper <pdf filename|stem>`: start from a specific paper. - `--repetitions <number>`: repeat every paper/mode combination. Model, runtime, and sandbox controls: - `--model <name>`: override the Codex model. - `--reasoning-effort <minimal|low|medium|high|xhigh>`: override reasoning effort. - `--max-turns <number>`: set the maximum main Codex cycles per paper for iterative modes. - `--max-concurrent-jobs <number>`: run independent paper/mode/repetition jobs in parallel. - `--page-render-dpi <number>`: preferred DPI when agents render selected PDF pages. - `--representative-output-frames <number>`: preferred number of Morpheus output frames to sample when visual inspection is needed. - `--codex-turn-timeout-ms <number>`: timeout for a single Codex turn. - `--codex-windows-sandbox <elevated|unelevated>`: Windows command-runner sandbox mode. - `--bridge-tool-timeout-ms <number>`: timeout for host-side Python bridge calls. - `--morpheus-run-timeout-ms <number>`: timeout for individual Morpheus executions. - `--morpheus-xml-schema-path <path>`: override the MorpheusML schema used for diagnostics. - `--no-analysis`: skip the automatic final `analyze_benchmark.py` step. Quota, recovery, and aggregation: - `--no-codex-quota-fallback`: fail immediately on Codex quota/rate-limit errors. - `--codex-quota-fallback-wait-minutes <number>`: fallback wait when no reset time is reported. - `--codex-quota-max-retries <number>`: maximum quota retries per turn. - `--codex-quota-retry-buffer-seconds <number>`: buffer added after parsed reset times. - `--recover-run <benchmark run dir>`: rebuild a benchmark summary from an existing run folder and resume eligible interrupted runs. - `--continue-incomplete`: allow recovery to continue incomplete or max-turn runs. - `--combine-runs <benchmark run dir> <benchmark run dir> [...]`: combine completed benchmark folders. - `--combined-run-dir <output dir>`: output folder for a combined run summary. Normal benchmark runs automatically run one recovery pass equivalent to `--recover-run <new run folder> --continue-incomplete` before returning the final summary. This catches interrupted or incomplete runs that still have remaining original cycle budget. When `--recover-run` is used manually, the target run's saved `benchmark_summary.json.config` becomes the default config. Explicit CLI flags still override it; for example, add `--max-turns 7` to give exhausted runs more cycles. Recovery also honors `--max-concurrent-jobs`, with the same Windows elevated-sandbox safety cap as normal benchmark runs. ## Analysis figures After a normal run, manual recovery, or run combination, the CLI runs the analysis script automatically by default. To regenerate analysis tables and figures manually, run: ```powershell python analyze_benchmark.py benchmark_runs/<timestamp>_benchmark ``` By default the script writes to `<benchmark_dir>/analysis_figures/`. Use `--out-dir <path>` to choose another output directory and `--formats png svg pdf` to control figure formats. Pass `--no-analysis` to `npm run benchmark -- ...` if you want the benchmark CLI to skip the automatic final analysis step. The analysis output includes: - `benchmark_analysis_by_run.csv` - `benchmark_analysis_by_mode.csv` - `benchmark_cost_quality_by_mode.csv` - `benchmark_attempt_self_scores.csv` - `benchmark_full_comparison` - `benchmark_scores_by_mode` - `benchmark_score_heatmaps` - `benchmark_cost_time_by_mode` - `benchmark_outcomes_efficiency` - `benchmark_cost_quality_pareto` - `benchmark_attempt_self_scores_by_paper` - `benchmark_attempt_self_scores_by_mode` Attempt-level plots use `attempt_reproduction.json` files written by agents inside attempt folders. If an agent attempt folder is missing that file, the analysis script derives the attempt number from the `attempt_###` directory name. Missing files in failed or log-only attempt folders are scored as 0; missing files in output-bearing attempt folders are kept in the CSV as unscored/blank values so intermediate successful attempts are not converted into artificial zero-quality points. ## Direct MCP usage The repository's supported MCP transport is **stdio**. The benchmark runner uses the same contract internally, and the repo-root `.mcp.json` exposes it for local MCP clients. Equivalent startup forms are: ```powershell python server.py ``` or via `.mcp.json`: ```json { "mcpServers": { "morpheus": { "command": "python", "args": ["server.py"] } } } ``` This repo does not currently advertise an HTTP `/mcp` endpoint as a supported integration surface. ## Runtime defaults The benchmark default is now `maxTurnsPerPaper = 5`. In the runner, a cycle is one main host-controlled Codex iteration for a paper. Morpheus attempts are executions of `run_morpheus_model(...)` and are counted separately. The cycle budget can be overridden on the command line: ```powershell npm run benchmark -- --max-turns 8 ``` Jobs run sequentially by default (`maxConcurrentJobs = 1`). To run independent paper/mode jobs in parallel, raise the cap: ```powershell npm run benchmark -- --all-benchmark-modes --max-papers 1 --max-concurrent-jobs 2 ``` This cap is also the practical upper bound on how many agents can start Morpheus work at the same time. On Windows, the benchmark defaults Codex command execution to `codexWindowsSandboxMode = "unelevated"` because elevated sandbox setup uses shared local sandbox accounts and can fail under parallel Codex jobs. Use `--codex-windows-sandbox elevated` or set `codexWindowsSandboxMode` in `benchmark.config.json` if stronger Windows isolation is more important than parallel throughput. The benchmark also gives Morpheus MCP tool calls, Python bridge calls, and Morpheus executions one-hour timeouts by default (`mcpToolTimeoutMs`, `bridgeToolTimeoutMs`, and `morpheusRunTimeoutMs` all default to `3600000`). The default MorpheusML schema validator is `src/benchmark/MorpheusML.xsd`. ## Reading the outputs Each invocation creates a master directory such as `benchmark_runs/20260427_153012_benchmark/`. Inside that directory: - `benchmark_summary.json` Aggregate result across processed papers, including total duration, token totals, and timing diagnostics. - `benchmark_manifest.json` Run-level diagnostics and a compact list of paper runs. - `benchmark_comparison.json` and `benchmark_comparison.csv` Mode-level comparison summaries across the selected capability modes. - `<timestamp>_<paper-stem>/` One isolated working directory per benchmark paper and mode. Codex runs with this paper directory as its working directory, so it does not see sibling paper outputs. Inside a paper run directory you will typically find: - `run_manifest.json` Paths and metadata for the staged inputs and run outputs. - `agent_workspace/` The isolated workspace used by the benchmark subject. This contains staged paper assets, agent-created scripts, agent attempt folders, and copied model artifacts. - `codex_home/` Snapshot of the isolated Codex home used for the subject run, including persisted session files needed for recovery. - `paper.txt` Extracted text from the benchmark PDF. - `paper_visual_manifest.json` A lightweight index of likely figure and table pages in the PDF. - `model.xml` The most recent Morpheus model produced by the agent. - `xml_versions/` Captured and harvested model XML revisions, including per-attempt `model.xml` copies when available. - `attempts/attempt_###/` Host-side canonical Morpheus attempt logs and generated outputs used for technical evaluation. - `agent_workspace/.../attempt_###/attempt_reproduction.json` Agent-authored attempt-level reproduction self-score, when the mode can execute Morpheus during the agent turn. - `technical_evaluation.json` Executability score and breakdown. - `reproduction_report.json` The structured reproduction rubric returned by the agent. - `transcripts/` Per-cycle Codex interaction logs. The benchmark diagnostics split total agent turn wall time into Morpheus simulation tool time and non-Morpheus agent analysis/tool time. They also report main cycles, Morpheus attempts, host Morpheus run time, and host technical evaluation time. Total benchmark wall time is recorded separately as `durationMs`. ## Benchmark behavior - The benchmark uses local Codex authentication. It is intended to run with a locally authenticated Codex CLI rather than direct API-key billing in the harness. - Live web search is available during benchmark runs when the agent genuinely needs outside information, but benchmark judgments should remain grounded in the staged paper assets, the local Morpheus reference material, and the Morpheus outputs generated in the run directory. - The agent is given paper text and a visual manifest up front. Paper page images are not attached by default; instead, agents render only the specific figure pages they need and inspect local PNGs directly with `view_image` when available. - If `benchmark_focus/<paper-stem>.txt` exists, the runner tells the agent to prioritize that target when the paper contains multiple models or figures. - Agents can also sample representative Morpheus output images and inspect those local PNGs directly. The runner no longer has separate image-review turns, image-review budgets, `outputImagePaths`, or `contactSheetPath` fields. - A cycle is one main host-controlled Codex iteration for a paper, not one tool call. Morpheus attempts are executions, whether launched by the agent through MCP or by the host for non-MCP modes. - For Morpheus-running modes, prompts ask agents to create a fresh attempt folder for each Morpheus attempt, copy the corresponding `model.xml` into that folder, inspect the outputs, and write `attempt_reproduction.json`. - The technical score is deterministic, but the reproduction report is a structured qualitative assessment returned by the agent from the staged evidence. - Aggregate average scores are final-available-model averages: every paper/mode run with a non-null final technical or reproduction score contributes, including runs that reached the maximum cycle budget. - Results can vary between runs because the benchmark depends on an LLM-driven modeling loop. In practice, a one-paper smoke test is the best first check, and repeated full runs should be interpreted as distributions rather than perfectly fixed outputs. - Codex quota/rate-limit errors are handled by the runner. When the CLI reports a reset time, the runner waits until that time plus `codexQuotaRetryBufferSeconds`; otherwise it waits `codexQuotaFallbackWaitMinutes` before retrying, up to `codexQuotaMaxRetries` per turn. Set `codexQuotaFallbackEnabled` to `false` or pass `--no-codex-quota-fallback` to fail immediately instead. ## Supported workflow vs. legacy material The supported workflow is the Codex SDK benchmark runner plus the Morpheus skill and thin MCP server in the repository root. Everything under `Archive/` is historical material from the earlier Anthropic-based setup. It is kept for comparison, not for routine use.