内容
# mcp-load-lab
A load and latency harness for **Model Context Protocol (MCP)** servers.
Most MCP tooling checks whether a server is *correct*: does it follow the spec,
is its tool surface stable, is it safe to deploy. `mcp-load-lab` answers a
different question: **how does the server behave under concurrent load?** It
drives real `tools/call` traffic at a chosen concurrency, measures latency
percentiles and throughput, and fails your CI build when a server misses its
performance budget.
It has **no required dependencies** for the stdio transport (Python standard
library only). HTTP transport adds a single optional dependency.
## Install
```bash
pip install mcp-load-lab # stdio transport, zero deps
pip install "mcp-load-lab[http]" # adds streamable-HTTP support (httpx)
```
Or from a clone:
```bash
pip install -e .
```
## Quick start
Every example below works against the bundled demo server in `examples/`,
so you can try the tool before pointing it at your own.
List a server's tools:
```bash
mcp-load-lab tools --stdio "python examples/echo_server.py"
```
Ad-hoc load run, no config file needed:
```bash
mcp-load-lab quick \
--stdio "python examples/echo_server.py" \
--tool sleep --arg ms=20 \
--concurrency 32 --duration 5 --warmup 1
```
```
phase conc reqs rps ok% mean p50 p90 p95 p99 max
------- ---- ---- ------ ----- ---- ---- ---- ---- ---- ----
warmup* 32 1485 1454.5 100.0 21.7 21.6 22.4 22.5 23.5 23.6
load 32 4512 1498.3 100.0 21.3 21.3 21.9 22.1 22.4 22.9
(latencies in ms; * = warmup, excluded from gates)
```
## Scenario files
For repeatable runs and CI, describe the whole test in a JSON or TOML file:
```jsonc
{
"name": "echo-sleep-ramp",
"transport": {
"type": "stdio",
"command": "python",
"args": ["examples/echo_server.py"]
},
"calls": [
{ "tool": "sleep", "arguments": { "ms": 20 }, "weight": 3 },
{ "tool": "echo", "arguments": { "text": "ping" }, "weight": 1 }
],
"phases": [
{ "name": "warmup", "concurrency": 4, "duration_s": 1, "warmup": true },
{ "name": "c8", "concurrency": 8, "duration_s": 3 },
{ "name": "c32", "concurrency": 32, "duration_s": 3 },
{ "name": "c64", "concurrency": 64, "duration_s": 3 }
],
"timeout_s": 10,
"gates": {
"max_p95_ms": 120,
"max_error_rate": 0.01,
"min_throughput_rps": 200
}
}
```
Run it:
```bash
mcp-load-lab run examples/scenario.json --json report.json
```
- **`calls`** are chosen per request by `weight` (weighted random). Use this to
model a realistic mix of tools.
- **`phases`** run in order. A `warmup` phase is measured but excluded from the
gates. Describe a concurrency ramp by listing several phases, each with a
higher `concurrency`.
- **`gates`** are optional thresholds. If any gate fails, the aggregate of all
non-warmup phases is reported as `FAILED` and the process exits non-zero.
## Transports
- **stdio** launches your server as a subprocess and speaks newline-delimited
JSON-RPC over its pipes. Concurrent requests are multiplexed over the single
connection with distinct ids, so a server that handles requests concurrently
shows real concurrency.
```jsonc
{ "type": "stdio", "command": "python", "args": ["server.py"], "env": {}, "cwd": "." }
```
- **http** POSTs JSON-RPC to a streamable-HTTP endpoint using a pooled client
(needs the `http` extra). Session ids returned on `initialize` are reused.
```jsonc
{ "type": "http", "url": "http://localhost:8000/mcp", "headers": {} }
```
## CI gating
`mcp-load-lab` exits `0` when all gates pass and `1` when any gate fails, so it
drops straight into a pipeline as a performance regression check:
```yaml
- name: MCP performance budget
run: mcp-load-lab run perf/scenario.json --json perf/report.json
```
Gates can also be set or overridden on the command line:
```bash
mcp-load-lab run perf/scenario.json \
--max-p95 150 --max-error-rate 0.01 --min-rps 300
```
## Metrics
Per phase and for the non-warmup aggregate the harness reports request count,
throughput (requests/second over wall-clock time), success rate, and latency
`min / mean / p50 / p90 / p95 / p99 / max` in milliseconds. Percentiles use
linear interpolation over the sorted per-request latencies. Failures are
broken out by kind: `tool_error` (server returned `isError`), `rpc_error`
(JSON-RPC error), `timeout`, and `exception`.
## Demo server tools
`examples/echo_server.py` is a dependency-free async MCP server exposing four
tools chosen to exercise different load shapes: `echo` (trivial), `sleep(ms)`
(I/O-bound), `fib(n)` (CPU-bound), and `flaky(rate)` (fails with a given
probability, for testing the error path and error-rate gates).
## Load model
The engine is **closed-loop**: each phase runs a fixed number of concurrent
workers, and each worker issues one call, waits for the response, and
immediately issues the next until the phase's `duration_s` or `requests` budget
is spent. This measures how a server responds at a specific concurrency level.
Open-loop (fixed target RPS) pacing is on the roadmap.
## License
Apache-2.0. See [LICENSE](LICENSE).
