mcp-vs-cli-bench

# MCP vs CLI Is the Wrong Fight This repository contains the benchmark suite behind Smithery's blog post, [MCP vs CLI Is the Wrong Fight](https://smithery.ai/blog/mcp-vs-cli-is-the-wrong-fight). The suite compares the same tasks and backends across four surfaces: - raw API priors - raw APIs with machine-readable specs - thin native MCP tools - generic CLI surfaces It is designed to measure agent experience, not to prove that one interface always wins. ## Current Headline Findings - Specs help raw API calling: **51.7%** to **73.3%** success in `api_priors_vs_specs`. - Thin native MCP beats direct API use with specs on the same tasks: **55.0%** to **91.7%** success in `specs_vs_native_mcp`. - On the same thin surface, MCP beats the described CLI: **91.7%** vs **83.3%** success in `native_mcp_vs_cli`. - On the full 826-tool GitHub catalog, explicit CLI search closes part of the gap: **66.7%** to **87.5%**, while native MCP remains at **100.0%**. ## Scope Experiment families: - `api_priors_vs_specs` - `specs_vs_native_mcp` - `native_mcp_vs_cli` - `cli_topology` - `linear_graphql_precision_ablation` - `github_large_catalog_native_mcp_vs_cli` - `github_large_catalog_cli_topology` - `github_large_catalog_search_affordance` Services: - GitHub REST - 24-operation curated slice - 826-tool frozen catalog for large-catalog discovery tests - Linear GraphQL - frozen slice for interface tests - live read-only rerun for the GraphQL appendix - public repo ships a redacted task template for the live workspace-specific prompts - Singapore Bus REST - 8-operation niche API slice Models: - Claude Code Haiku 4.5 - Claude Code Sonnet 4.6 for the large-catalog GitHub experiments - Codex GPT-5.4 The checked-in public artifacts are sanitized: - `bench/results/results.jsonl` contains only the declared 732-run matrix. - `bench/config/tasks/linear.yaml` keeps the task structure but redacts workspace-specific strings. ## Quick Start ```bash python -m venv .venv . .venv/bin/activate pip install -e ".[dev]" ``` Common commands: ```bash make preflight make matrix make smoke make pilot make full make report ``` Useful docs: - Method: [`docs/method.md`](docs/method.md) - Benchmark guide: [`docs/benchmark-guide.md`](docs/benchmark-guide.md) - Current report: [`docs/report.md`](docs/report.md)