mcp-replay

# mcp-replay -brightgreen)   Record real MCP (Model Context Protocol) traffic once, then replay it against a new build of your server to catch behavioral regressions before they ship. Every other quality check on an MCP server tests it with inputs you generate: schema linters read the tool surface, fuzzers throw malformed payloads, load harnesses measure latency. None of them tell you whether your server still *behaves* the way it did last week. `mcp-replay` captures what your server actually returned in a real session and locks it in as a contract, so a change in behaviour shows up as a failing check even when the tool schema is untouched. Stdlib only. No runtime dependencies. Python 3.9+. ## Why this exists A static surface diff catches a renamed tool or a widened schema. It cannot catch a server that quietly starts returning the wrong number, drops a field from a result, or flips a success into an error while keeping the same schema. That is exactly the class of regression `mcp-replay` is built for. ``` static surface -> mcp-surface-diff (did the schema change?) valid-input contract -> mcp-conformance-kit (does it follow the spec?) adversarial input -> mcp-fuzz (does bad input crash it?) performance -> mcp-load-lab (is it fast enough?) context cost -> mcp-context-budget (is the surface cheap?) behaviour over time -> mcp-replay (does it still do the same thing?) ``` ## Install ```bash pip install mcp-replay # once published # or, from a clone: pip install -e . ``` You can also run it without installing: `python -m mcp_replay ...`. ## 30-second demo ```bash mcp-replay demo ``` This records a session against a bundled sample server, then replays it against a healthy build and against a regressed twin whose tool surface is identical but whose behaviour has drifted: ``` == Replay against the HEALTHY server (expect grade A, no regressions) == [PASS] #0 initialize [PASS] #2 tools/list [PASS] #3 tools/call [PASS] #4 tools/call [PASS] #5 tools/call GRADE A (100.0/100) == Replay against the REGRESSED server (same tool surface, changed behaviour) == [PASS] #0 initialize [PASS] #2 tools/list [FAIL] #3 tools/call structure-change $.result: key 'echoed' added [FAIL] #4 tools/call value-change $.result.content.0.text: '5' -> '6' [PASS] #5 tools/call GRADE B (80.0/100) ``` `tools/list` is byte-for-byte identical between the two servers, so a static surface diff sees nothing. `mcp-replay` still catches the two behaviour changes. ## Record a real session `record` is a transparent stdio proxy. Point your MCP client at `mcp-replay` in place of the server, and it forwards everything both ways while teeing request/response pairs into a cassette. For Claude Desktop: ```json { "mcpServers": { "my-server": { "command": "mcp-replay", "args": [ "record", "--server", "python /path/to/my_server.py", "--cassette", "/path/to/session.jsonl", "--redact", "session_id" ] } } } ``` Use the client normally. Every tool call flows through and lands in the cassette. Secrets in known keys (`authorization`, `api_key`, `token`, and so on) plus bearer tokens and long opaque strings are redacted before they touch disk; add more keys with `--redact`. ## Replay in CI ```bash mcp-replay replay \ --server "python /path/to/my_server.py" \ --cassette session.jsonl \ --min-grade B \ --fail-on-regression ``` `--fail-on-regression` exits non-zero if any regression or missing response is found. `--min-grade` exits non-zero below the given letter. Wire either into a GitHub Actions step to block a merge that changes behaviour. Output formats: default console, `--json`, `--markdown`, and `--badge FILE` which writes a shields.io endpoint JSON you can publish as a status badge. Tune the comparison when a field is legitimately volatile: - `--ignore-path result.content[].text` ignores a response path entirely. - `--allow-reorder` treats a list with the same members in a different order as a match rather than a regression. Timestamps and UUIDs are masked automatically, so a server that stamps every result with the current time still scores a clean match. ## Regression kinds | Kind | Meaning | | --- | --- | | `missing-response` | server returned nothing or crashed on replay | | `error-status-change` | a result became an error, or the reverse | | `structure-change` | a key was added or removed, or a value's type changed | | `ordering-change` | a list has the same members in a different order | | `value-change` | a leaf value differs | ## Grade A 0..100 score over four weighted dimensions, mapped to A-F: | Dimension | Weight | Meaning | | --- | --- | --- | | Responsiveness | 25% | every request got a well-formed response | | Match rate | 45% | normalized response matched the recording | | Error parity | 20% | success/error class matched the recording | | Structural stability | 10% | no keys added/removed, no type flips | Any missing response caps the grade at D, because a server that stops answering is not "mostly stable". ## Cassette format JSONL. The first line is a meta header; each following line is one interaction. ```json {"schema": "mcp-replay/cassette@1", "meta": {"server": ["python", "my_server.py"]}} {"seq": 0, "kind": "request", "ts": 0.0, "message": {"jsonrpc":"2.0","id":1,"method":"initialize","params":{}}, "response": {"jsonrpc":"2.0","id":1,"result":{}}} {"seq": 1, "kind": "notification", "ts": 0.0, "message": {"jsonrpc":"2.0","method":"notifications/initialized"}, "response": null} ``` Notifications are recorded so the handshake replays faithfully. They are sent fire-and-forget on replay and are not asserted. ## Scope (v1) Focuses on the client-to-server request/response flow over stdio, which covers tool calls, listing, and the initialize handshake. Server-initiated messages (notifications, sampling requests) are forwarded during recording but not asserted on replay. HTTP/SSE transports are not yet supported. ## Development ```bash python -m unittest discover -s tests -v python -m mcp_replay demo ``` ## License MIT. See [LICENSE](LICENSE).