consumer-agent-testing-tool

<p align="center"> <img src="docs/assets/catt-logo.svg" width="320" alt="CATT logo: an orange cartoon cat at a desk, smiling over its shoulder, with CATT spelled in code on the monitor"> </p> <h1 align="center">Consumer Agent Testing Tool (CATT)</h1> Consumer Agent Testing Tool (CATT) tests how an assistant uses your service. Run a scripted or model-driven agent against an MCP server, an MCP app, a website or an HTTP API. Check what the service did, inspect the evidence, and compare results across builds. **Muse is the default emulation and testing target.** CATT models Meta's Muse through a local test host. **Instinct is an experimental target** for the connector lane and its supported browser, HTTP, scheduling and presentation extensions. See the [Instinct spec and research](docs/instinct-emulation.md) for scope and limits. The model that drives a trial is separate from the assistant platform being emulated. [Contributing](CONTRIBUTING.md) | [Security](SECURITY.md) | [Changelog](CHANGELOG.md) | [MIT license](LICENSE) > **A test host.** CATT is a surrogate: it approximates an assistant's > host, not its private implementation. A passing test does not certify > compatibility with Muse or any hosted assistant. Each Muse-specific behavior > has a recorded basis: `documented`, `unofficial`, `assumed` or `unknown`. > This project is not affiliated with or endorsed by Meta, Instinct, OpenAI or Anthropic. ## What you can test today | Lane | Command | What it checks | Status and scope | | --- | --- | --- | --- | | MCP connectors | `connector-run SCENARIO` | Use the server's tools, apply account and approval rules, protect private values, and check the service's recorded outcome and the agent's answer. | Muse surrogate; experimental Instinct target with `--emulation-target instinct`. See [connector lane](docs/connector-lane.md). | | Connector extensions | `connector-run SCENARIO` with `extensions:` | Routes, memory, custom API clients, browser actions, wallet cards, background jobs, untrusted content, channel limits and simulated users. | Supported with the lane. `user_simulator` is exploratory and cannot run in scripted tests or decision contracts. See [extensions](docs/connector-extensions.md). | | Scenario scaffolding | `scaffold URL --output FILE`, `scaffold --from-run RUN --output FILE` | Read a tool catalog or turn an agent run into scripted calls. Calls no service tools. Replace every `TODO(scaffold)` before running. | Supported. One Streamable HTTP connector; no OpenAPI input. | | Muse connector readiness | `check URL --suite muse-guidelines` | Review the catalog and optional manifest against the pinned connector guidelines without calling tools. | Documented constraints, not admission. See [readiness checks](docs/readiness-checks.md). | | Personal Agent Protocol | `check ORIGIN --suite poppy` | Read `poppy.json`. With a manifest, probe sessions, sign-in and one declared operation on the company's hosts. | Draft 0.1; no production Muse evidence. | | Muse gadget registration | `check FILE --suite gadget` | Check a `link.register` message and command specs against the pinned SDK limits. Opens no connection. | Documented limits, not admission. The SDK's terms say it is not a developer platform. | | MCP protocol contracts | `check TARGET` | Check schemas, version negotiation, sessions and operation contracts; run the pinned conformance suite. | Supported. Generic MCP evidence; see [readiness checks](docs/readiness-checks.md). | | Embedded MCP apps | `surface-run SCENARIO` | Render the app in Chromium and check forms, tool calls, resources and service responses. An agent driver chooses from the scenario's scripted steps. | Supported. Generic MCP Apps host, not Muse. See [host scope](#what-the-mcp-apps-host-implements). | | Websites | `browse POLICY`, `run --surface browser` | Act on accessibility snapshots under request rules. Takeover pauses the run (exit 3); the CLI has no handback. | Native Chromium, without OS isolation. Browser scenarios use only the packaged reservation portal. | | HTTP APIs | `run --surface api-tools`, `inspect`, `check` | Bind approval to each proposed call and check backend outcomes. | Supported with a mapping. Your API also needs a loopback deployment and private control adapter. See [API scenarios](docs/api-scenarios.md) and [API trials](docs/api-behavior.md). | | Evidence | `report`, `export`, `replay`, `diff` | Inspect reports, redact exports, verify playback and compare runs. | Supported. Playback and diff require sealed runs. | | Development extensions | `--development-extensions COMMAND` | CLI binaries, isolated browsers, cloud workers, local Glimmer, generated OpenAPI and Prism mocks. | Outside the supported release. See [development extensions](#development-extensions). | The connector lane records the basis for each Muse-specific behavior. Tool classes and approval prompts follow documented guidelines. The `muse-cli` projection and template client follow unofficial mirrors; the native projection is an assumed generic baseline. Instinct runs have a separate source record and label the shared host algorithms as assumptions. Native MCP is CATT's test adapter; no public Instinct MCP integration contract was found. Run the local booking example under Instinct without a model or account: ```sh uv run consumer-agent-testing-tool connector-run examples/connector/instinct-booking.yaml \ --driver scripted --script examples/connector/fixture-booking.calls.json \ --start "python -m consumer_agent_testing_tool.connector_fixture --port {port} --variant good --auth none" ``` Set `emulation_target: instinct` in your scenario or pass `--emulation-target instinct`. Instinct uses native projection and the generic MCP client. It defaults to plain-text replies and does not run Muse guideline checks. Its trusted-network protocol, Vault, payment rails and Files are not implemented; the [spec](docs/instinct-emulation.md) records those gaps. Three quickstarts follow. The [connector lane](#quickstart-connector-lane) and the [popular MCP server](#quickstart-a-popular-mcp-server) quickstarts run on loopback with no account; the [Coverage Cat sandbox](#quickstart-coverage-cat-sandbox) quickstart calls a hosted sandbox. [Test your own service](#test-your-own-service) turns them into a workflow, and the [documentation index](docs/README.md) lists every page. ## Setup You need Python 3.12 or newer (CI tests 3.12 on Linux and 3.14 on macOS), [uv](https://docs.astral.sh/uv/), and Node.js 20 or newer with npm (the installed packages declare `>=20`; tested with 25.9). The recorded release evidence covers macOS. From a clone of this repository, install the Python and Node dependencies and the Chromium build the browser lanes use: ```sh uv sync --extra openrouter --extra dev npm --prefix tooling ci --ignore-scripts npm --prefix tooling/payment-ui ci --ignore-scripts PLAYWRIGHT_BROWSERS_PATH="$PWD/.cache/ms-playwright" node tooling/payment-ui/node_modules/playwright/cli.js install chromium ``` The connector lane, the readiness checks and the scripted runs need only `uv sync`. Node and Chromium are for the MCP Apps and browser lanes. On Linux (Debian or Ubuntu), install Chromium with its system libraries instead. Linux is not covered by the recorded release evidence: the commands below match the [CI workflow](.github/workflows/ci.yml) but have not been qualified here. ```sh sudo apt-get update && sudo apt-get install -y nodejs npm # or Node 20+ from nodesource or nvm curl -LsSf https://astral.sh/uv/install.sh | sh uv sync --extra openrouter --extra dev npm --prefix tooling ci --ignore-scripts npm --prefix tooling/payment-ui ci --ignore-scripts PLAYWRIGHT_BROWSERS_PATH="$PWD/.cache/ms-playwright" \ node tooling/payment-ui/node_modules/playwright/cli.js install --with-deps chromium ``` Three features rely on macOS Seatbelt (`sandbox-exec`) and report unsupported or blocked on Linux rather than running unconfined: the `--offline` readiness mode, the network-denied validator workers behind `run --surface api-tools`, and the local-model memory profile. The connector lane, the scripted runs, the readiness checks and the Docker-backed stdio targets do not depend on it. Check the setup with `doctor`: ```sh uv run consumer-agent-testing-tool doctor ``` It checks Node against the floors the installed packages declare, the `tooling/payment-ui` packages against their pinned versions, and Chromium and its headless shell under `.cache/ms-playwright`. It reports the optional protocol tooling, and checks Docker only for development lanes or configured stdio MCP and CLI targets. It makes no model requests, even when an existing configuration or `--model-profile` selects a model. Each failed or blocked check prints one line on stderr with its reason and, for a setup problem, the command that fixes it. To explicitly run the provider preflight, which sends two short model requests and may incur charges, add `--model-preflight`: ```sh uv run consumer-agent-testing-tool doctor --config agent.yaml --model-preflight ``` Scripted runs need no model account, and the scripted commands in all three quickstarts need no configuration file. Agent runs need a model profile and an API key; [Model providers](#model-providers) shows how to create one. Supply credentials through environment variables, and keep them out of scenarios and shared evidence. `consumer-agent-testing-tool.yaml`, the default settings file, is local and git-ignored, and no profile is shipped. ## Quickstart: connector lane This quickstart stays on loopback and needs no account. Complete [Setup](#setup), then start the Cedar Stays-like test service in a second terminal. It prints one JSON line with its URLs when it is ready; Ctrl-C stops it: ```sh uv run python -m consumer_agent_testing_tool.connector_fixture --port 8765 --variant good --auth none ``` Replay the example's scripted calls, with no model: ```sh uv run consumer-agent-testing-tool connector-run examples/connector/fixture-booking.yaml \ --driver scripted --script examples/connector/fixture-booking.calls.json ``` The script searches rooms, gets a quote, books and gives a final answer. The host fills the guest's email from the scenario's `private:` values, which the model never sees, and the scripted user answers each approval card from the scenario's `permissions:` rules. After the run, the host checks the fixture's private oracle (one confirmed booking, no duplicate charges) and the booking's status through `get_booking`. It also checks that each sent call matched its approval card (`approval.dispatches_match`, `approval.disclosure_bound`) and that the final answer made no false claim and relayed the total and the confirmation (`answer.no_false_claims`, `answer.relay_complete`). The command prints the run directory and `Connector report: .../report.html`, and exits 0 when every required check passed. Runs are sealed under `runs/connector-<id>/`. Share the output of `export`, not the run directory: its `private/` folder holds host-private values. For an agent run, create `agent.yaml` with the `init` command in [Model providers](#model-providers), set `OPENROUTER_API_KEY` in your shell, and run: ```sh uv run consumer-agent-testing-tool connector-run examples/connector/fixture-booking.yaml --config agent.yaml ``` The model preflight runs first and prints its own evidence path. The agent then sees the four allowed tools as native functions, with the host-private email removed from their schemas, and writes the other arguments itself. This uses paid model inference. To change the treatment without editing the scenario, `--projection muse-cli` offers the same tools through one `exec` command, `--exposure search-only` loads their parameters through `tool_search`, `--output-schema strict` refuses results that break a tool's `outputSchema` as the official MCP SDK does, and `--trials N` runs independent trials and prints pass^k. The run lock records the projection, exposure, client-profile and output-schema overrides under `overrides`. `connector-run` exits 0 when every required check passed; 1 when a check failed, the answer made a false claim or an agent limit was reached; 2 for a setup, harness or reporting error; and 3 when the trial is blocked or unsupported, for example by an uncertain write to reconcile, an unobservable check or a client-profile limit. To start from your own server, run `scaffold`; to replay a recorded, shareable agent trial without a model, turn it into a calls file with `scaffold --from-run`. The [connector lane](docs/connector-lane.md) reference covers scenarios, flags and evidence. The [examples](examples/connector/README.md) list each shipped scenario with its fixture command, and [lane extensions](docs/connector-extensions.md) covers routes, custom connectors, the browser sub-agent, wallets, schedules and the other extensions. ## Quickstart: a popular MCP server The connector lane works with any MCP server over Streamable HTTP, not only the bundled test service. This quickstart points it at "Everything" ([`@modelcontextprotocol/server-everything`](https://github.com/modelcontextprotocol/servers/tree/main/src/everything), MIT), the reference server the MCP project maintains for client authors. Its weather tool returns simulated data, so it needs no account or key. Start it in a second terminal; the first run downloads the pinned release from npm: ```sh PORT=3001 npx -y @modelcontextprotocol/server-everything@1.0.0 streamableHttp ``` See what the host infers from the live catalog. `scaffold` runs the handshake and `tools/list`, calls no tool, and writes a scenario skeleton with suggested tool classes from the server's annotations: ```sh uv run consumer-agent-testing-tool scaffold http://127.0.0.1:3001/mcp --output my-everything.yaml uv run consumer-agent-testing-tool check http://127.0.0.1:3001/mcp --suite muse-guidelines ``` The second command reviews the catalog statically against the pinned Muse connector guidelines. Then replay the shipped example, which asks for the weather in Chicago, first with native tools and then through the Muse-style command-line projection over the unofficial Muse template client: ```sh uv run consumer-agent-testing-tool connector-run examples/connector/everything-weather.yaml \ --driver scripted --script examples/connector/everything-weather.calls.json uv run consumer-agent-testing-tool connector-run examples/connector/everything-weather.yaml \ --projection muse-cli --client-profile muse-template-unofficial-2026-10 \ --driver scripted --script examples/connector/everything-weather-cli.calls.json ``` Both exit 0. The scenario offers the model three of the server's tools; a reviewed allowlist keeps the rest off its list, including `get-env`, which returns the server process's environment variables, and `gzip-file-as-resource`, which fetches any URL it is given. The server has no private oracle, so the outcome check calls `get-structured-content` itself after the agent stops, and the answer checks look for the temperature and conditions the server returned. With a model profile, let an agent do the same task three times and print pass^k (paid inference). The scenario declares no reset URL, so `connector-run` warns that the trials share the server's state; this weather tool keeps none: ```sh uv run consumer-agent-testing-tool connector-run examples/connector/everything-weather.yaml \ --config agent.yaml --trials 3 ``` ## Readiness checks `check` also runs readiness suites that read catalogs and documents instead of running an agent. With the connector-lane quickstart's fixture still running, review its catalog and the example manifest against the pinned Muse connector guidelines: ```sh uv run consumer-agent-testing-tool check http://127.0.0.1:8765/mcp --suite muse-guidelines \ --manifest examples/connector/readiness.yaml ``` | Suite | Command | Exit codes | | --- | --- | --- | | `muse-guidelines` | `check URL --suite muse-guidelines [--manifest FILE]` | 0 no fail finding, 1 fail findings, 2 setup error, 3 the client profile cannot reach the server | | `muse-guidelines` drift | `check --suite muse-guidelines --drift-check` | 0 unchanged, 1 drifted, 2 not fetched | | `poppy` | `check ORIGIN --suite poppy [--manifest FILE]` | 0 no fail finding, 1 fail findings, 2 setup error, 3 blocked | | `gadget` | `check FILE --suite gadget` | 0 no fail finding, 1 fail findings, 2 the document cannot be read | | `contract` (default) | `check TARGET [--interface-profile PROFILE]` | 0 every required check passed, 1 a check failed, 2 setup, harness or reporting error, 3 blocked or unsupported | The guidelines review lists the server's tools and calls none; `--drift-check` fetches the live guidelines, and only when given. Poppy discovery only reads. Its probes run only with `--manifest` and act on the company's own hosts: they start sessions, sign in the declared test account and confirm one declared operation. The gadget suite reads one local file. The contract suite checks a target named in the configuration file (`--config`). For a `kind: mcp` target without declared `contract_cases`, its `operation-contracts` check is unsupported and the run exits 3. A clean readiness result is not an admission. Meta reviews connectors by hand, Poppy is draft 0.1 with no production Muse evidence, and the Gadget SDK's terms say it is not a developer platform. [Readiness checks](docs/readiness-checks.md) describes the rules, manifests, `--offline` behavior and the `mcp-contract-2026-07-28` profile. ## Quickstart: Coverage Cat sandbox From a source checkout or unpacked source distribution, complete [Setup](#setup), then run: ```sh uv run consumer-agent-testing-tool surface-run examples/coveragecat-sandbox.yaml --driver scripted ``` The example creates a fictional shopper's draft in Coverage Cat's hosted **sandbox** at `https://www.coveragecat.com/api/consumer/mcp`. It needs no Coverage Cat account, operator credentials or model API key. The first `umbrella_consumer_prefill` call sets `sandbox: true`, as specified in the [consumer MCP manifest](https://www.coveragecat.com/.well-known/mcp.json). That argument activates the server's sandbox. The scenario's `test_only: true` flag merely records the test author's intent. The host fixes the tool arguments and keeps the returned access token private. It permits two tools, `umbrella_consumer_prefill` and `consumer_intake_issues`, and checks that: 1. Prefill returns `sandbox: true`. 2. Follow-up retains `sandbox: true`. 3. Follow-up returns the original application ID. 4. The server reports `application_submitted: false`. Each run creates a new draft and stops before submission. A failed or missing sandbox check stops the workflow. These checks cover draft creation and follow-up, not payment widgets, browser behavior or independent backend checks. To let a model choose the steps, create `agent.yaml` with the `init` command in [Model providers](#model-providers), set `OPENROUTER_API_KEY` in your shell, and run: ```sh uv run consumer-agent-testing-tool surface-run examples/coveragecat-sandbox.yaml \ --driver agent --config agent.yaml ``` The agent driver is a **scripted action menu**: Spark Open picks one ready scenario step per turn, and the host runs it with the scenario's arguments. The model never sees the server's tools, schemas or raw results, only the result fields the scenario exposes, and the evidence is labeled `scripted action menu`. The host supplies the test inputs and checks the results. This version uses paid model inference through Together, with ZDR enabled and fallback disabled by default. **Verified 15 October 2026:** both scripted and Spark Open agent runs passed all four sandbox checks. Both model responses identified Together; both requests used the configured ZDR policy. See the [recorded quickstart results](docs/release-evidence/coveragecat-sandbox-quickstart-2026-10-15.json). Each command prints an outcome line, such as `Surface run completed (exit 0)`, then `Surface report: .../report.html`. Open the report to inspect the workflow and its checks. The same directory contains `result.json` for the structured result and `events.jsonl` for the redacted event timeline. Agent runs also store model and tool records under `agent/`. Replace the placeholders below with the run directories printed by your commands: ```sh uv run consumer-agent-testing-tool report runs/surface-YOUR_RUN_ID uv run consumer-agent-testing-tool export runs/surface-YOUR_RUN_ID uv run consumer-agent-testing-tool diff runs/surface-BASELINE_ID runs/surface-CANDIDATE_ID ``` Share the generated export. Keep the full run folder private. While a run can resume, `checkpoint.private.json` holds the credentials, tool results and host inputs it needs; a completed run, or one that failed with exit 1 and left no write unconfirmed, keeps only a scrubbed record. To test your own MCP service, copy the example and replace its endpoint, allowed origins, tool policies, test data and response checks. [Surface workflows](docs/surface-workflows.md) explains UI steps, the agent driver and resuming runs. ### Extend through submission and sandbox checkout The [checkout example](examples/coveragecat-sandbox-checkout.yaml) adds review, submission, Monoline offer selection, consent, bind follow-up, document upload and simulated payment-method setup. Its 14 steps contain 62 assertions. All seven tools use the consumer umbrella MCP, and every service step must return the same sandbox intake. **Current integration status, 15 October 2026: blocked at submission.** Coverage Cat returns quote fields that its MCP output schema excludes. It also returns `minimum_auto_limits` as a string where the schema requires an object. The host rejects the response. The draft quickstart passes, but the full checkout test has **not passed end to end**. See the [failure report and separate diagnostics](docs/coveragecat-sandbox-checkout.md). Run this command to reproduce the failure or retest after the service contract is corrected: ```sh uv run consumer-agent-testing-tool surface-run examples/coveragecat-sandbox-checkout.yaml \ --driver scripted \ --replies examples/coveragecat-sandbox-checkout-replies.json ``` The reply files supply a **fictional shopper's answers**: approval to submit, credit consent, the Monoline $2M offer at $389, sufficient auto limits, declined excess UM/UIM, and “No” to both carrier questions. These fixtures do not authorize transactions for a real shopper. A changed quote or missing or negative approval stops the next write. The two generated declarations PDFs are labelled `SYNTHETIC TEST ONLY`. Once submission works, the example pauses with exit code 3 at `sandbox_document_review_poll`. **Only at that checkpoint**, wait at least the reported 60 seconds and resume the printed run: ```sh sleep 60 uv run consumer-agent-testing-tool surface-run examples/coveragecat-sandbox-checkout.yaml \ --driver scripted --resume runs/surface-YOUR_RUN_ID \ --replies examples/coveragecat-sandbox-checkout-after-review.json ``` The host pauses for your reply but does not measure how long you wait. The after-review fixture approves the simulated payment callback. For an agent-driven run, replace `--driver scripted` with `--driver agent --config agent.yaml` on **both** commands. A resumed run must keep its original scenario, driver and model configuration. The sandbox supplies mock payment credentials. The example sends the returned `seti_sandbox_...` with the fixture `pm_sandbox_consumer_agent_testing_tool`, then requires `payment_method_on_file`, `waiting_on_carrier`, and no charge confirmation. Billing checks inspect setup metadata rather than rendered fields. No Stripe form opens, no card details are collected, and no fallback URL is used. This test establishes neither premium collection nor active insurance. Test the embedded payment UI and Stripe test-mode setup separately. ## Test your own service The quickstarts above are the pieces of one workflow. For an MCP server you own: 1. **Write a scenario.** `scaffold URL --output my-connector.yaml` reads the live catalog (`--auth-env VAR` for an API key, or `--auth-from FILE` with a scenario that holds an OAuth block) and writes a skeleton. Replace every `TODO(scaffold)`: the user's goal in their own words, the allowlist of tools a reviewed connector would expose, tool classes your documentation settles, approval rules for the scripted user, host-private values such as the user's email, and checks that read the real outcome, from a private state URL (an oracle) or one of your read tools. 2. **Check readiness.** `check URL --suite muse-guidelines --manifest FILE` reviews the catalog and your declarations against the pinned connector guidelines. A clean result is a documented constraint, not an admission. 3. **Pin the happy path.** Replay a scripted calls file with `--driver scripted`; it needs no model and belongs in CI. After an agent trial you trust, `scaffold --from-run RUN --output FILE` turns it into one. 4. **Run agent trials.** `--trials N` with a model profile gives pass^k. Run the same scenario under each treatment a Muse-like host may use: `--projection muse-cli`, `--exposure search-only`, `--client-profile muse-template-unofficial-2026-10` and `--output-schema strict`. The run lock records each one. 5. **Add realism.** [Lane extensions](docs/connector-extensions.md) add competing connectors and connect-on-request, payment with single-use cards, background jobs, untrusted content, channel limits and a simulated user. 6. **Compare builds.** `diff` compares two sealed runs and exits 1 on an outcome regression. Share the output of `export`; the run directory's `private/` folder holds host-private values. For an MCP app with its own UI, start from the Coverage Cat scenario and `surface-run`. For a website, use `browse` with a browser policy, or the connector lane's `browser` extension. For an HTTP API, write a mapping for `run --surface api-tools`, or test whether an agent can build its own client from your docs with the `custom_connector` extension. [Run a consumer test](#run-a-consumer-test) covers the other lanes. ## Model providers Create a profile with `init`. This one uses **Muse Spark Open** (`meta/muse-spark-open`) through OpenRouter, routed only to Together, with zero data retention (ZDR) required and fallback disabled; set `OPENROUTER_API_KEY` in your shell before an agent run: ```sh uv run consumer-agent-testing-tool init --provider openrouter --model meta/muse-spark-open \ --routing-provider together --config agent.yaml ``` `init` also stages, beside the configuration, the API scenarios, the browser scenarios and the mapped-API example. It stages only scenarios the supported CLI can run, and refuses providers that have only a configuration template (`openai`, `anthropic`, `gemini`, `azure-openai`, `vertex-gemini`, `openai-compatible`) unless `--development-extensions` comes before `init`. `init --provider none --config scripted.yaml` writes a configuration for scripted commands that need one. `init` refuses to overwrite an existing file. Profiles that `init` writes are labeled `specified/unverified`. `doctor --model-preflight`, `run` and the agent drivers of `connector-run`, `surface-run` and `browse` print a warning for any profile not labeled `supported`, and the run lock records the label. Model IDs that contain `contributor` select a mode in which the provider may train on your prompts; such a profile needs `allow_training_on_prompts: true` (`init --allow-training-on-prompts`). See [model providers](docs/model-providers.md) for every adapter, its checks and its failure labels. ### OpenRouter model and privacy settings Consumer Agent Testing Tool requests **zero data retention (ZDR) by default**, even when `routing.zdr` is omitted. To change this, edit the selected profile in `consumer-agent-testing-tool.yaml` or the file passed to `--config`: ```yaml model_profiles: openrouter-dev: provider: openrouter api: chat-completions model_id: meta/muse-spark-open auth: mode: api_key env: OPENROUTER_API_KEY routing: only: [together] allow_fallbacks: false require_parameters: true data_collection: deny zdr: true # Set to false to disable this tool's per-request ZDR requirement. defaults: model_profile: openrouter-dev ``` Use unquoted YAML booleans: `true` or `false`. Consumer Agent Testing Tool applies the setting to every OpenRouter request, including requests that continue after a tool result, and records it in the routing evidence. Setting `zdr: false` removes this request-level requirement. It leaves `data_collection: deny`, the pinned provider and the ban on fallbacks in place. Your OpenRouter account or guardrails may still enforce ZDR. Request logs show the policy sent; they cannot independently verify the provider's retention practices. See [OpenRouter's ZDR documentation](https://openrouter.ai/docs/guides/features/zdr). **Checked 15 October 2026:** OpenRouter listed Together for Spark Open in both its model and ZDR endpoint catalogs. A live tool-call test passed with the routing settings above. Both responses identified Together, and the model used the tool result correctly. See the [test record](docs/release-evidence/together-spark-zdr-live-2026-10-15.json). To choose another provider, set `routing.only` or pass `--routing-provider` to `init`. Requests never switch providers or models silently. Runs use Inspect's dedicated `openrouter` provider, which replays earlier reasoning in structured form rather than as text. Each response must name the pinned provider. Responses do not name endpoint variants, so a pin such as `nebius/fp4` is recorded as `variant unverified`. Streaming is not supported. See [model providers](docs/model-providers.md#openrouter). With `zdr: true`, only ZDR-listed endpoints are eligible. On 15 October 2026, Muse Spark 1.3 and Muse Wonder were served only by Meta on OpenRouter and were not ZDR-listed, so no ZDR-compliant OpenRouter route existed for them. A request for them is expected to fail, but no live run has recorded the error. When OpenRouter's error cites the data policy, it is labeled `no endpoint satisfies the routing/ZDR policy`. With `routing.only: [together]`, the `init` default, it may instead be labeled `routing policy: no allowed provider`. Sources: [Meta announcement](https://research.meta.ai/blog/muse-spark-open), [OpenRouter model](https://openrouter.ai/meta/muse-spark-open), [provider endpoints](https://openrouter.ai/api/v1/models/meta/muse-spark-open/endpoints), and [ZDR endpoints](https://openrouter.ai/api/v1/endpoints/zdr). The standalone smoke test and older consumer helper read the same `--config` and `--model-profile`, including privacy settings. Profiles that `init` writes leave reasoning effort unset. ```sh .venv/bin/python scripts/openrouter_smoke.py --config agent.yaml \ --model-profile openrouter-dev ``` This command tests the model connection with synthetic tool data. It needs an API key and an available route to the pinned provider. It makes no requests to the service you plan to test. ### Meta Model API `init --provider meta` writes a profile for Meta's Model API: ```sh uv run consumer-agent-testing-tool init --provider meta --model muse-spark-1.3 --config meta.yaml ``` The profile uses the Responses API at `https://api.meta.ai/v1` and reads the key from `MODEL_API_KEY` or `META_API_KEY`; if both are set, they must agree. Add `--api chat-completions` for Chat Completions, or `--key-env NAME` to use another variable, which is then the only one read. The adapter uses the OpenAI SDK from the `openrouter` extra. The host refuses requests that Meta rejects, such as a forced tool choice, before sending them (exit 3). Meta adds steering context to every prompt on its servers, so captured requests are incomplete, and every Meta run records this as an observation gap. Streaming is not supported, and the API is pay-as-you-go. See [Meta Model API](docs/model-providers.md#meta-model-api). No live Meta Model API run has been recorded; the adapter is tested against local fixtures. ## Run a consumer test This section covers the embedded MCP apps (`surface-run`), browser (`browse`) and HTTP API lanes. To let an agent use your MCP server's own tools, use the [connector lane](#quickstart-connector-lane). A scenario defines the endpoint, allowed origins, tools, private arguments, UI actions, consent requirements and result checks, and optionally an API key for the MCP connection (`auth`). Use endpoints and synthetic transactions you have permission to test. ```sh consumer-agent-testing-tool surface-run path/to/scenario.yaml --driver scripted consumer-agent-testing-tool surface-run path/to/scenario.yaml --driver agent --config agent.yaml consumer-agent-testing-tool surface-run path/to/scenario.yaml --resume runs/existing-run consumer-agent-testing-tool browse path/to/browser-policy.json --actions path/to/actions.json ``` The host declares MCP Apps support as a client, finds each tool's UI through its `tools/list` definition and renders it in a native browser. With `--driver agent`, the model picks among the scenario's steps (a scripted action menu) and never sees the server's tools, raw results or private inputs. Service checks determine the result; a rendered form or a model's claim of success cannot prove that a transaction completed. `browse` drives a restricted browser through accessibility snapshots and element references, with no selectors, scripts or CDP. Its policy file sets the start `url`, the permitted `origins` and request `rules`, `snapshot_max_chars` (default 20,000), `history_snapshots` (2) and `dialogs` (`agent`, `accept` or `dismiss`), as in [examples/browser-policy.json](examples/browser-policy.json). Long snapshots are paged with `{"kind": "snapshot", "offset": N}`. Refused or failed actions come back to the agent as observations with a `status`, and dialogs and file choosers are answered with `kind: dialog` or `kind: upload`. With `--driver agent`, the provider preflight runs before the browser opens. The result's `observations` block counts truncated snapshots and refused actions. When the agent requests a takeover, the run pauses (exit 3). The CLI has no handback, so control does not return to the agent in that run. In native mode, WebSockets opened by dedicated workers and WebRTC TCP relays are not enforced. The policy field `webmcp: true`, off by default, lists page-registered WebMCP tools in the snapshot and lets the agent call them with `kind: webmcp`; that lane is labeled "generic WebMCP page tools, not Muse" (basis: `assumed`), and its evidence notes that Meta documents WebMCP for its AI glasses, a Meta product that is not Muse. See [restricted browser behavior](docs/release-extensions.md#restricted-browser-behavior). HTTP API tests use explicit mappings with the scenario runner. Mappings can send declared request headers and put the target credential in a custom API-key header. In API and browser trials, invalid tool proposals are returned to the model as tool errors, and exhausted limits fail the trial; see [API scenarios](docs/api-scenarios.md#agent-mistakes-and-limits). To test whether an agent can build its own client from an API's documentation instead, use the connector lane's `custom_connector` extension. Prism, Schemathesis and generated-OpenAPI tools are available only as development extensions. Their five packages with unresolved original license notices are excluded from the supported release, not treated as cleared. ### Outcomes and exit codes `surface-run` follows the exit codes in spec §18: | Exit | Meaning | | --- | --- | | 0 | Completed | | 1 | Failed: an assertion, a service rejection, a UI interaction failure or an agent error | | 2 | Setup, infrastructure, integration or reporting failure | | 3 | Blocked or unsupported: waiting for a reply or approval, an unconfirmed write, a limit, or missing coverage | Before `Surface report: ...` it prints one line: `Surface run <status> (exit N) at <stage> [<class>]: <reason>`. `result.json` records the same `failure_class` and `failure_stage`, and whether the run is `resumable`. A completed run, or one that failed with exit 1, is final unless the failure left a write unconfirmed: that write may have taken effect, so the run stays resumable to reconcile it. Blocked runs and setup failures can resume. `run` and `browse` use the same four codes, except that exhausting a turn, tool-call or time limit is a failure (exit 1). Only Inspect operator and custom limits, and in `run` a deadline that passed before an attempt's first generation, block instead (exit 3). `run` prints each evidence path on stdout. `browse` prints the final observation and its evidence path when the run completes or pauses for takeover, and nothing on stdout otherwise. For any outcome other than completed, both also print one line on stderr: `<label> <status> (<class>) at <check>: <reason>. Evidence: <path>`. The label names the step, for example `Model preflight`, `Trial 2/10 of stale_total`, `Resumed trial of stale_total` or `Browse` (`doctor` uses `Doctor check`). A `run` with a failed or blocked trial, or with `--trials` above 1, also prints a summary line with the counts. With `--trials` above 1 it adds the suite pass^k, the mean over scenarios, for example `Run completed: 0 failed, 0 blocked and 4 completed of 4 trials; pass^1 1.00, pass^2 1.00. Evidence: ...`. The suite's metrics also break the trials down by scenario ([statistical decisions](docs/statistical-invariants.md)). `run DIR` refuses (exit 2), before the preflight, a directory whose scenario files share an id. `connector-run` prints the same stderr line, labeled for example `Connector trial 1/1 of fixture-booking-happy-path`; its codes are listed in the [connector-lane quickstart](#quickstart-connector-lane). With `--trials N`, one code stands for all trials: 2 when any trial's evidence is not shareable, else 1 when any trial failed, else 2 when any hit a setup, harness or verification error, else 3 when any was blocked, else 0. The `Connector trials:` line on stderr gives pass^k and names the trials that are not shareable or ended in an error. The readiness suites' codes are in [Readiness checks](#readiness-checks). ### What the MCP Apps host implements The embedded-app lane is generic MCP Apps host evidence (MCP Apps 2026-01-26), for hosts such as ChatGPT or Claude. It is not Muse evidence: Muse documents no third-party UI. Each `result.json` says so under `mcp_apps_host`. The host implements: - the client capability (`io.modelcontextprotocol/ui` with `text/html;profile=mcp-app`), UI discovery from the tool's `tools/list` definition, and HTML resources as text or base64 blobs; - `tool-input` with the call's arguments, `tool-result`, and a `hostContext` (theme, display mode, locale, time zone, platform, container width and maximum height, and tool information); - a content security policy built from the resource's `_meta.ui.csp`, or the specification's restrictive default when the resource declares none, and the declared camera, microphone, geolocation and clipboard-write permissions; - in the native renderer, a sandbox proxy on an origin separate from the host page. With `browser_origins`, the host page also has Chromium's local-network-access permission, every redirect hop is checked again (GET or HEAD to an allowlisted origin only), and each direct write is recorded as an unconsented `host_action` before it is sent; - view `tools/call` requests for the step's allowlisted tools, and `resources/read` and `resources/list` for the step's `callback_resources`; - `ui/message`, delivered once to agent runs in the next observation a model reads, as `app_messages`; the view is told when the host does not deliver it. `ui/update-model-context`, `ui/open-link`, display-mode, log and teardown requests are recorded, and the latest model-context update reaches agent runs as `app_context`; - size changes, `ui/resource-teardown` before the view closes, and `tool-cancelled` when the view never received the call's result; - `_meta.ui.visibility`, for view calls and for tool steps; - MCP 2026-07-28 servers, when the scenario sets `version_negotiation: auto`. The default is the 2025 `initialize` handshake; - an API key on the MCP connection (`auth`), as a bearer token, a custom header or a query parameter; - a legacy `window.openai` bridge. It does not implement a model turn for a view's message, resource templates, `_meta.ui.domain`, display modes other than inline, partial tool input, host-context change notifications, OAuth on the MCP connection, or consent for a view's direct writes in the native renderer. The legacy bridge and the isolated renderer have no sandbox proxy. See [MCP Apps host](docs/surface-workflows.md#mcp-apps-host). ## Inspect retained evidence ```sh consumer-agent-testing-tool report runs/existing-run consumer-agent-testing-tool export runs/existing-run consumer-agent-testing-tool replay runs/existing-run consumer-agent-testing-tool diff runs/baseline runs/candidate ``` Replay checks the evidence seals and stops on a mismatch. It never substitutes a live request. To test the service again, start a new run. Reports mark missing measurements as unknown and keep partial totals distinct from complete totals. Reports open with the verdict, which names the failing step and check, then the checks and host capabilities, and a per-step timeline that shows what the model received in full. Then come a connectors table for runs with several connectors, the surface views (what the service published, what the model was offered, what it received at each step, and what only the developer sees), the metrics, and the viewers and linked evidence: agent and preflight runs, their Inspect logs and the `inspect view` command, and, locally, a captured Playwright trace. Gaps and limits follow, then the run lock and the full recorded result, each collapsed; see [report layout](docs/report-recovery.md#report-layout). `report` on an agent sub-run of a sealed run returns its sealed report unchanged. Evidence files and exports hide credentials, registered secrets, contact details and other personal fields such as names and addresses. Model views keep the user's own data visible: they hide only credentials and declared secrets, plus browser password fields and, in a resumed browser scenario, text typed in an earlier attempt. The provider boundary refuses any model request that still contains a registered credential. Export also applies the run's own private values, which are never exported, and refuses (exit 2) when a staged file, or a member of an Inspect log or other archive, still holds one. See [redaction](docs/sealed-event-provenance.md#redaction). `diff` compares two sealed runs, such as surface, browser or connector runs. A change to the package source counts as a run-lock change, so compare runs made from the same checkout. It exits 0 when they are comparable with no outcome regression, and 1 when an outcome regressed: a worse status, or a passing check that no longer passes or is missing. It exits 3 when they are not comparable, because of a run-lock change outside `--treatment` or an unsealed or unverifiable run. Generated IDs and service timestamps are normalized; run locks are compared exactly. `--ignore POINTER` (repeatable) excludes a JSON pointer under `/outcomes`, `/capabilities` or `/milestones`, and `--treatment /server` declares an intentional server build change. See [replay and diff](docs/release-extensions.md#replay-and-diff). For more detail, see the [documentation index](docs/README.md), [connector lane](docs/connector-lane.md), [lane extensions](docs/connector-extensions.md), [readiness checks](docs/readiness-checks.md), [consumer tests](docs/consumer-test.md), [browser diagnostics](docs/browser-diagnostics.md), [measurement coverage](docs/measurement-coverage.md), [model timing](docs/model-timing.md), and [release extensions](docs/release-extensions.md), which includes development history. ## Use it in CI Scripted connector runs need no model and no key, and they replay the same calls every time, so they work as a pull-request check. `connector-run` can start the service itself and write reports a CI system reads: ```sh uv run consumer-agent-testing-tool connector-run my-connector.yaml \ --driver scripted --script my-connector.calls.json \ --start "python -m my_service --port {port}" \ --junit reports/connector.xml --summary-json reports/connector.json ``` - `--start COMMAND` runs the service on a free loopback port. The command is split like a shell command but not run in a shell. - `{port}` in the command is that port. The service also gets it as `PORT` and the scenario as `${PORT}`. - The tool waits until `--ready-url`, or the scenario's first connector endpoint, answers HTTP. It waits at most `--ready-timeout` seconds (default 30). - When the run ends, the service's process group gets SIGTERM, then SIGKILL after 5 seconds. - Its output goes to `runs/service-<id>.log`, outside the sealed run. That log isn't redacted, so pass secrets to the service through the environment. The run lock records the command with credentials redacted. - A service that exits early or never becomes ready exits 2. - `--junit FILE` writes one test suite per trial and one test case per check. Blocked checks are `<skipped>`. `check --suite muse-guidelines` accepts it too. - `--summary-json FILE` writes the verdict, exit code, checks, trials, pass^k and run paths. Its format is [`schemas/summary-v1.schema.json`](schemas/summary-v1.schema.json). `--summary-json -` prints only that JSON on stdout and moves the other lines to stderr. - Both reports are also written when the command stops with an error. See [CI outputs and services](docs/connector-lane.md#ci-outputs-and-services). The exit codes in [Outcomes and exit codes](#outcomes-and-exit-codes) decide the step. A 3 (blocked or unsupported) is not a pass. [`examples/ci/connector-lane.yml`](examples/ci/connector-lane.yml) is a GitHub Actions template with two jobs. On pull requests, it runs the scripted calls and the readiness review. Nightly or on demand, it runs agent trials under each Muse-like treatment, and only when a model key secret is set. [`.github/workflows/ci.yml`](.github/workflows/ci.yml) runs this repository's tests and shipped examples on macOS and Linux. Coding agents should start with [`AGENTS.md`](AGENTS.md). All matrix jobs feed the required `CI` check; Linux failures are not allowed to pass silently. Workflow validation, tracked-file hygiene, secret scanning and schema freshness are checked too. ## Running the tests Plain `pytest` runs the supported collection: ```sh uv run pytest -q ``` `tests/conftest.py` gives each session a temporary directory, home and cache under `.cache/pytest/`, removes them after a passing session, and removes the surface and browser run directories its tests create under `runs/`. The full collection takes about twenty minutes and makes no network request beyond loopback and no model request. `pytest --development-extensions` also collects the experimental tests, which may need other runtimes, tools and external evidence. Async cleanup and unhandled thread exceptions fail tests instead of remaining warnings. See [CONTRIBUTING.md](CONTRIBUTING.md) for the review workflow and [the launch checklist](docs/open-source-launch.md) for the private-to-public transition. ## Release audit The supported release is **`local-consumer-v1`**: the connector lane and its extensions, the readiness checks, consumer MCP apps, the restricted browser, mapped HTTP APIs, transactional consent, continuation, evidence and replay, measurements, the distribution, and the OpenRouter and Meta Model API adapters. Real-Muse calibration, cloud provisioning, local Glimmer, browser OS isolation and live Bedrock qualification are outside it; see [release scope](docs/release-scope.md) for the full boundary. Release evidence is generated per release and is not committed. [`docs/release-gates.json`](docs/release-gates.json) is the committed policy: the gates, the scope and the exclusions. The [release workflow](.github/workflows/release.yml) runs on a `v*` tag. It records every gate on macOS, requires the audit to pass, and attaches three things to the GitHub release: - the wheel and the source archive, with signed build provenance; - `release-evidence-<commit>.tar.gz`, which holds `release/release-gates.json` (the recorded ledger) and `release/evidence/`. Use `consumer-agent-testing-tool release-audit` to check release readiness. It reads `release/release-gates.json` when evidence has been recorded or extracted there. Without it, it reads the committed policy and reports every gate unverified. Exit code 0 means **release-ready for `local-consumer-v1`**; exit code 3 means at least one required gate needs attention. The evidence binds a digest of the source, so a source change or an expired gate (seven days after recording) cannot inherit an earlier passing result. To audit a published release, check out its tag, extract its evidence archive at the repository root and run the audit before the gates expire. Day to day, you don't need any of this. Run the tests for what you change, and let [CI](.github/workflows/ci.yml) run the full collection on each pull request. A separate live Spark Open test passed through Together with ZDR enabled. It verified native tool calls, continuation after a tool result, and the requested routing policy. It did not test streaming, independently verify retention practices, or establish compatibility with every consumer workflow. The Coverage Cat checkout failure described above remains an open integration issue. The audit requires four gates: | Gate | Evidence required | | --- | --- | | Regression | Every test in the default supported collection passes, with no skips | | Distribution | Fresh offline installation, original dependency notices, artifact integrity and packaged consumer workflows | | Transactional consent | Passing consent, continuation and effect checks in the current regression | | Replay/diff | Passing evidence-integrity and comparison checks in the current regression | To record evidence locally instead, for example before tagging, run the same steps as the workflow, in this order. `verify_distribution.py` regenerates the dependency notices, which are part of the source fingerprint, so the distribution comes before the regression. The regression takes about twenty minutes. Everything is written under `release/`, which git ignores: ```sh .venv/bin/python scripts/verify_distribution.py .venv/bin/python scripts/record_distribution.py .venv/bin/python scripts/run_regression.py release/evidence/regression-run.xml .venv/bin/python scripts/record_regression.py release/evidence/regression-run.xml .venv/bin/python scripts/record_release_tests.py release/evidence/regression-run.xml consumer-agent-testing-tool release-audit ``` `run_regression.py` refuses an existing report path, so remove `release/` or pick a new name to record again. It records workspace-relative paths and writes the home directory as `~` in the log. Distribution checks use caches in the workspace and the installed browser. They need no Docker VM, cloud worker or model credentials. If a dependency cache is missing, populate it separately with `scripts/verify_distribution.py --prepare-cache`; that setup step uses the network. The installation evidence covers macOS only. The audit verifies source fingerprints, artifact hashes, required checks and expiry dates. It returns 0 when every required gate passes, or 3 otherwise. Excluded capabilities remain labelled as excluded. [Qualification inputs](docs/qualification-inputs.md) explains what each gate needs. ## Development extensions The repository retains worker, isolation, local-model, calibration and generated-OpenAPI experiments, and the CLI/skill lane for a developer's own CLI binary (`run --surface cli-skill`), which needs Docker and an oracle that returns the reservation fixture's snapshot schema. The normal CLI rejects these paths. To use them, put `--development-extensions` before the subcommand and install the separate `development-extensions` dependency group and any required tooling. These experiments are outside the supported release. The supported way to give an agent a CLI over an MCP server is `connector-run --projection muse-cli`. The optional Node API tools have [known dependency advisories](docs/dependency-security.md). Do not use them with untrusted inputs while those findings remain open. The [original specification](docs/spec-v0.3.md) and earlier test records document the project's history; they do not define the scope of this release. ## License MIT; see [LICENSE](LICENSE). The license does not cover the Coverage Cat and CATT names or the logo; see [TRADEMARKS.md](TRADEMARKS.md). Third-party components keep their own licenses; the distribution ships their notices under `third_party/`.