agent-otel

# `agent-otel` + `scry` > **Agent-native observability, in two layers.** > `agent-otel` — the OTel-native router + sinks + replay (the substrate). > `scry` — the SDK and CLI an agent uses to query its own traces. 🚧 v0.0.20 — pre-alpha, APIs may change. MIT. --- `agent-otel` is the substrate: declarative fanout to any number of backends, replay for retroactive rerouting, reversible PII masking. App engineers wire it up the same way whether the consumer is a human, an agent, or both. `scry` is where the agent-first thesis lives: an SDK and CLI for an agent to inspect its own traces in-process or from a shell. Think `kubernetes` + `kubectl` — library and CLI, dual-named on purpose. Phoenix/Braintrust/Langfuse render traces for humans; `scry` gives an agent a query surface over the same data. They compose. ## Install ```bash npm install agent-otel # or: bun add agent-otel ``` `scry` ships as a CLI in the same package: ```bash npx scry --help # after bun add agent-otel: bunx scry --help ``` ## 60-second install — Anthropic / OpenAI + Braintrust If you're on `@anthropic-ai/sdk` or `openai` and Braintrust today, this is the whole setup: ```ts import Anthropic from '@anthropic-ai/sdk'; import { NodeSDK } from '@opentelemetry/sdk-node'; import { defineRouter } from 'agent-otel'; import { instrument } from 'agent-otel/anthropic'; import { braintrust, postgres } from 'agent-otel/sinks'; import { withPrivacy, PrivacyProxy } from 'agent-otel/privacy'; // 1. Wire the router → backends const proxy = new PrivacyProxy(); const router = defineRouter({ sinks: { braintrust: withPrivacy( braintrust({ apiKey: process.env.BRAINTRUST_API_KEY!, project: 'support-agent' }), { proxy, redactKeys: ['auth.token'] }, // PII never reaches Braintrust ), archive: postgres({ url: process.env.DATABASE_URL! }), // your own escape hatch }, rules: [{ match: '*', to: ['braintrust', 'archive'] }], }); new NodeSDK({ spanProcessors: [router.asSpanProcessor()] }).start(); // 2. Wrap your client. That's it. Every call now emits a perfect // OpenInference span — to Braintrust (masked) AND your archive (raw). const anthropic = instrument(new Anthropic({ apiKey: process.env.ANTHROPIC_API_KEY! })); const resp = await anthropic.messages.create({ model: 'claude-sonnet-4-7', max_tokens: 1024, messages: [{ role: 'user', content: 'Hello' }], }); ``` **For OpenAI, swap two lines:** ```ts import OpenAI from 'openai'; import { instrument as instrumentOpenAI } from 'agent-otel/openai'; const openai = instrumentOpenAI(new OpenAI({ apiKey: process.env.OPENAI_API_KEY! })); const resp = await openai.chat.completions.create({ model: 'gpt-5.5', messages: [{ role: 'user', content: 'Hello' }], }); ``` Same auto-instrumentation. Built-in cost tables for GPT-5.5 / GPT-5 / GPT-4.1 / GPT-4o families; date-pinned model IDs (`gpt-5.5-2026-04-23`) match by prefix. What you get for those ~10 lines: - ✅ Every Anthropic / OpenAI call traced with OpenInference attributes (gen_ai.\*, llm.\*, tool calls flattened) - ✅ Braintrust dashboards work as before (real evals, real playground), **but with PII masked** - ✅ Your own Postgres archive — query with `scry trace tree <id>` from the CLI - ✅ Real production cost and token counts on every span (Sonnet/Opus/Haiku tables built in) - ✅ Replay any stored call counterfactually (`replayLLMCall`) without re-running your whole agent E2E tested against the real Anthropic + Braintrust APIs (`tests/e2e/instrument-anthropic.test.ts`, `tests/e2e/privacy-braintrust.test.ts`). For OpenAI, Vercel AI SDK, Mastra, CrewAI — auto-instrument adapters are next. Today: use OpenInference's per-vendor packages alongside agent-otel's router (they emit OTel; we route OTel; same wire format). ## Router — one OTel emit, declarative fanout ```ts import { NodeSDK } from '@opentelemetry/sdk-node'; import { defineRouter } from 'agent-otel'; import { phoenix, braintrust, slack, jsonl } from 'agent-otel/sinks'; const router = defineRouter({ sinks: { phoenix: phoenix({ endpoint: process.env.PHOENIX_ENDPOINT, apiKey: process.env.PHOENIX_API_KEY }), braintrust: braintrust({ apiKey: process.env.BRAINTRUST_API_KEY!, project: 'support-agent' }), alerts: slack({ webhookUrl: process.env.SLACK_WEBHOOK_URL! }), archive: jsonl({ path: './traces.jsonl' }), }, rules: [ // Everything → archival { match: '*', to: ['archive'] }, // LLM calls → both eval platforms { match: { 'gen_ai.system': '*' }, to: ['phoenix', 'braintrust'] }, // Expensive LLM calls → ping #ai-cost-watch in Slack { match: { 'llm.cost.total': '>1.0' }, to: ['alerts'] }, // Errors → Slack AND Braintrust (so eval picks them up) { match: { 'status_code': 'ERROR' }, to: ['alerts', 'braintrust'] }, ], }); const sdk = new NodeSDK({ spanProcessors: [router.asSpanProcessor()] }); sdk.start(); ``` Your existing `tracer.startSpan(...)` calls now fan out per the rules. Add a sink, drop a sink, change a threshold — config-only, no app-code changes. ## `scry` — SDK and CLI for agents to query their own traces Two paths depending on where your agent runs. Pick one or use both: | Where the agent runs | Reach for | Looks like | |---|---|---| | **In-process** (Node.js / Bun, Vercel AI SDK, Anthropic SDK directly) | The **TypeScript SDK** | `sink.findSpans({ status_code: 'ERROR' })` from inside the same process the agent runs in | | **In a sandbox shell** (E2B, Daytona, your own Docker, anywhere with bash) | The **`scry` CLI** | `scry query --status=ERROR | jq` piped through standard shell tools | Both surfaces operate on the same data (an `Inspectable` sink — `memory` and `postgres` today, others can implement it). Pick whichever fits where the agent actually lives. ### Programmatic (in-process SDK — most agent code) For agents running in your Node/Bun process. Import the primitives directly; no shell, no JSON round-trip. Use this when your agent is a function in a TypeScript codebase calling LLMs/tools. ```ts import { memory } from 'agent-otel/sinks/memory'; import { and, substring } from 'agent-otel/filters'; import { buildTree, causalChain, renderTree } from 'agent-otel/trace-tree'; const sink = memory(); // ... router emits into sink ... const errors = sink.findSpans( and({ status_code: 'ERROR' }, substring('name', 'tool.')), { limit: 20 }, ); const tree = buildTree(sink.getTrace(traceId)); console.log(renderTree(tree, { attrs: ['llm.cost.total'] })); ``` ### MCP server (any MCP-aware agent: Claude Code, Cursor, Devin, …) Run `scry mcp` and any MCP client gets a tool surface for trace inspection. Local-dev pattern is to wire it into your client config so it spawns as a subprocess on demand: ```json // .claude/settings.json (Claude Code) — Cursor / Devin / etc. take similar config { "mcpServers": { "scry": { "command": "npx", "args": ["scry", "mcp", "--db", "postgres://localhost/myapp"] } } } ``` Or against a remote scry HTTP endpoint (in-sandbox / org-wide setups where the JWT was minted server-side): ```json { "mcpServers": { "scry": { "command": "npx", "args": ["scry", "mcp", "--endpoint", "https://api.example.com/v1/scry", "--token", "$SCRY_TOKEN"] } } } ``` Tools registered: - **`scry_query_jobs`** — list recent agent jobs (filter by status / attribute) - **`scry_get_trace`** — render a trace as an ASCII tree by `trace_id` - **`scry_causal_chain`** — walk root → target span path - **`scry_stats`** — aggregate counts / cost / duration / errors The same primitives that power the CLI and the SDK, exposed over MCP. First MCP server in LLM-trace-land. ### CLI (sandbox shell — and dev terminals) For agents running in a sandbox shell (E2B, Daytona, etc.) AND for human engineers debugging from a laptop. Composes with shell tools naturally. **Connect via direct DB or remote endpoint:** ```bash export SCRY_DB=postgres://localhost/mydb # direct Postgres (local / dev) # or for remote: export SCRY_ENDPOINT=https://scry.example.com export SCRY_TOKEN=<jwt> ``` Flags `--db`, `--endpoint`, `--token` work per-call too. **Three one-liners:** ```bash # Find all ERROR spans in the last 10 minutes, extract span IDs scry query --status=ERROR --since=10m --output=json | jq '.[] | .spanId' # Render the full call tree of a job (LLM ↔ tool ↔ DB) as ASCII scry trace tree 0123abcd... # Aggregate cost, latency, error rate across a filter scry stats --attr=gen_ai.system=anthropic ``` Full subcommand reference: ``` scry query [--status=X] [--kind=X] [--name=X] [--attr=k=v] [--since=10m] [--limit=N] scry trace get <trace_id> scry trace tree <trace_id> [--attrs=k1,k2] scry chain <trace_id> <span_id> # walk a span back to root: what led to this error? scry stats [--status=X] [--attr=k=v] ``` Composes naturally with shell tooling: `scry query --output=json | jq`, `scry stats | awk '$1 > 0.1 {exit 1}'`. No MCP boot, no ceremony. --- ## Cost tracking — `agent-otel/cost` Standard contract + OTel-native attributes for LLM cost. Three primitives: ```ts import { calculateCost, recordLLMCall, extractors, type PricingSource, } from 'agent-otel/cost'; // You supply pricing — agent-otel ships none. (Tables go stale; we don't // want to be the maintainer.) See examples/pricing-static.ts. const myPricing: PricingSource = { lookup: (m) => MY_TABLE[m] }; // In your provider stream's finish handler: const usage = extractors.openai(chunk.usage); // → cross-provider LLMUsage const cost = calculateCost('gpt-5', usage, myPricing); // → { cost, costType, breakdown } recordLLMCall(span, { usage, cost }); // → dual-write attrs ``` The span now carries **both** OpenInference (`llm.cost.total`, `llm.token_count.*`) **and** OTel-GenAI (`gen_ai.cost.total`, `gen_ai.usage.input_tokens`) — Phoenix, Arize, Langfuse, scry, Datadog GenAI, and any future OTel-GenAI consumer read it identically. ### Why "bring your own pricing"? Pricing data goes stale weekly (new models drop, providers re-price prompt cache). Baking a table into agent-otel would mean a release every time. We ship the **contract** — you wire a static map, fetch from [models.dev](https://models.dev), pull LiteLLM's JSON, hit your billing DB, whatever. See `examples/pricing-*.ts`. ### What you get on the span | Attribute | Both writes (OpenInference + OTel-GenAI) | |---|---| | Uncached input tokens | `llm.token_count.prompt`, `gen_ai.usage.input_tokens` | | Output tokens | `llm.token_count.completion`, `gen_ai.usage.output_tokens` | | Cache read / write | `llm.token_count.prompt_details.cache_read/write` | | Reasoning tokens | `llm.token_count.completion_details.reasoning` | | Total USD cost | `llm.cost.total`, `gen_ai.cost.total` | | Per-bucket breakdown | `gen_ai.cost.input/output/cache_read/cache_write/reasoning` | | Provenance | `gen_ai.cost.type` — `'actual' \| 'estimated' \| 'unknown'` | Provider-reported actuals (OpenRouter exposes `usage.cost`) are preferred over table-based estimates — set on `LLMUsage.provider_cost` by the OpenAI extractor when present. ### Provider extractors Pure functions, raw provider chunk → `LLMUsage`. Each knows the provider's quirks (OpenAI subtracts `cached_tokens` from `prompt_tokens`; Anthropic's `input_tokens` is already uncached; Gemini uses `cachedContentTokenCount`). ```ts extractors.openai(chunk.usage) // OpenAI / OpenRouter / Azure OpenAI extractors.anthropic(message.usage) // Claude / Bedrock Anthropic extractors.gemini(response.usageMetadata) ``` ## What this replaces You're probably doing some of these by hand right now: - **Phoenix SDK** for traces. **Braintrust SDK** for evals. **Datadog OTLP** for APM. **Sentry SDK** for errors. - A custom Slack/Discord script that scrapes logs for "LLM call > $X" alerts. - A nightly export script that copies a sample of traces to JSONL for fine-tuning later. - An ad-hoc adapter that reformats traces when you change eval vendors. `agent-otel` collapses all of that into one OTel emit + a declarative routing config. Same wire format everywhere; backends are just sinks. ## `agent-otel` × [OpenInference](https://github.com/Arize-ai/openinference) Different layers in the same pipeline: | | OpenInference | `agent-otel` | |---|---|---| | **Lives at** | SDK boundary (input) — wraps the LLM SDK so calls emit spans | Export boundary (output) — routes spans to sinks | | **Wraps** | Specific LLM SDKs | Nothing — consumes any OTel emitter | | **Replay / cost-aware sampling** | No | Yes | Use both: OpenInference makes your Anthropic SDK calls emit a span; `agent-otel` decides that span goes to Phoenix + Slack but not Braintrust. Convention compatibility is covered above. ## Convention modes — OpenInference, OTel GenAI, or both `agent-otel`'s instrumentations emit LLM telemetry under two parallel attribute name sets: - **OpenInference** (`llm.*`, `openinference.span.kind`, flattened messages) — production-mature, what Phoenix renders natively. - **OTel GenAI semantic conventions** (`gen_ai.*`, `gen_ai.operation.name`, structured messages) — the official direction; client spans went stable in early 2026, agent/tool spans still Development. Three modes, env-var-controlled — matches OpenTelemetry's [`OTEL_SEMCONV_STABILITY_OPT_IN`](https://opentelemetry.io/docs/specs/semconv/general/attribute-naming/#opt-in-attributes) pattern used for the HTTP semconv migration. Reusing the OTel-canonical env var means ops folks recognize it; no Daslab-specific flags. | Mode | Env var | What gets emitted | When to use | |---|---|---|---| | `openinference` | `OTEL_SEMCONV_STABILITY_OPT_IN=openinference` | OpenInference only (`llm.*`, flat messages) | Phoenix-only stack, no OTel-GenAI-native consumer | | `dup` (default) | unset or `gen_ai_dup` | Scalars in both name sets; content stays flat OpenInference | **Recommended.** Mixed environments. Phoenix still renders, OTel-native backends pick up `gen_ai.*` for free. | | `gen_ai` | `OTEL_SEMCONV_STABILITY_OPT_IN=gen_ai_latest_experimental` | `gen_ai.*` scalars only; content attributes skipped until structured `gen_ai.input.messages` JSON lands in a follow-up | Greenfield, OTel-native backends, willing to skip content until structured emission lands | Programmatic override for per-client control: ```ts import { instrument } from 'agent-otel/anthropic'; const client = instrument(new Anthropic({...}), { conventionMode: 'dup' }); ``` ### Why `dup` is the default Scalar dual-emit adds ~1-2 KB per span — negligible against the 64 KB raw-request blob already emitted. In exchange, the same span renders cleanly in Phoenix *and* shows up as a GenAI span in Honeycomb / Langfuse / DataDog with no flag set. The library serves both audiences out of the box. ### Content attribute pollution — why content stays flat The cheap part of dual-emit is scalars (model, tokens, finish reasons — ~10 attributes, ~1-2 KB). The expensive part is content: re-emitting full message bodies as `gen_ai.input.messages` JSON while still keeping flattened `llm.input_messages.N.*` would double per-span payload for message content. OTel's own guidance is to gate content attributes on opt-in; we currently keep content flat-OpenInference across `openinference` and `dup` modes. Structured `gen_ai.input.messages` JSON for `gen_ai` mode lands in a follow-up — for now, `gen_ai` mode emits scalars only and content attributes are skipped. ### Deprecation cadence When OTel GenAI agent/tool/eval spans are stable and Phoenix natively renders OTel GenAI attributes well, the default flips from `dup` → `gen_ai` (major-version bump). OpenInference remains supported via `OTEL_SEMCONV_STABILITY_OPT_IN=openinference` for two more minor releases, then dropped. Expected window: 6-12 months. Until then, `dup` carries no breaking changes. ## `agent-otel` × [OTel Collector](https://opentelemetry.io/docs/collector/) Both route OTel data; different runtimes and different audiences. - **Collector** is a Go sidecar configured in YAML — canonical for traditional APM with 100+ exporters in contrib. - **agent-otel** is a TypeScript library you `npm install` — agent-aware (knows `gen_ai.*`, `llm.cost.total`), ships sinks for eval/training platforms (Phoenix-as-dataset, Braintrust experiments, OpenPipe) the Collector doesn't have, and the wire format is the same. Run them alongside. They don't compete. ## `agent-otel` × Braintrust / Phoenix / Langfuse / LangSmith `agent-otel` sits in front of these — your eval/observability backend keeps its job; PII masking, a vendor-neutral archive, programmatic agent-side query, and replay layer on top. For an existing Braintrust user (same pattern works for Phoenix / Langfuse / LangSmith): ```ts import { defineRouter } from 'agent-otel'; import { jsonl, postgres, braintrust } from 'agent-otel/sinks'; import { withPrivacy, PrivacyProxy } from 'agent-otel/privacy'; const proxy = new PrivacyProxy(); const router = defineRouter({ sinks: { // KEEP: Braintrust as your eval/playground/experiments backend. // ADD: PII masking so customer emails / tokens never reach Braintrust. braintrust: withPrivacy( braintrust({ apiKey: process.env.BRAINTRUST_API_KEY!, project: 'support-agent' }), { proxy, redactKeys: ['auth.token'] }, ), // ADD: vendor-neutral local archive — escape hatch + audit trail archive: postgres({ url: process.env.DATABASE_URL!, table: 'spans' }), // ADD: cheap on-disk dump for backfill / replay later jsonl: jsonl({ path: './prod-traces.jsonl' }), }, rules: [{ match: '*', to: ['braintrust', 'archive', 'jsonl'] }], }); ``` What this adds on top of Braintrust: | Need | Braintrust alone | + agent-otel | |---|---|---| | Eval / playground / experiments | ✓ | ✓ (unchanged) | | Trace ingest + dashboards | ✓ | ✓ (unchanged) | | **PII masking before vendor sees it** | ✗ | ✓ via `withPrivacy()` (e2e tested against live Braintrust API) | | **Vendor-neutral archive** (Postgres / S3 / JSONL) | ✗ | ✓ | | **Programmatic agent self-debug** (`scry` SDK + CLI) | ✗ | ✓ | | **Counterfactual replay** ("what if Sonnet 4.7?") | manual playground only | `replayLLMCall()` — see below | | **MCP server** for Claude Code / Cursor to query traces | ✗ | planned | | **Lock-in escape** — leave whenever | hard | trivial; spans archived in your own store | Keep what works, add what's missing. ## Replay — retroactive routing The unique capability `agent-otel` unlocks: **change your mind about where spans go AFTER you've collected them.** Routing is configuration, not code, so the destinations aren't baked in at emit time. ```ts import { replay, fromJsonl } from 'agent-otel/replay'; await replay({ source: fromJsonl('./prod-traces.jsonl'), router: defineRouter({ sinks: { braintrust: braintrust({...}) }, rules: [{ match: '*', to: ['braintrust'] }], }), }); ``` Take spans you already captured, re-route them through any router config. Concrete workflows this enables: ### Customer debugging without touching prod (the daily-driver use case) A customer pings you: *"my agent broke yesterday at 3:14pm."* ```ts await replay({ source: fromJsonl('./prod.jsonl'), where: s => s.traceId === 'trace_xyz', router: defineRouter({ sinks: { slack: slack({ webhookUrl: DEBUG_CHANNEL }) }, rules: [{ match: '*', to: ['slack'] }], }), }); ``` Every step of that one trace pings you in Slack with attributes pretty-printed. Pure forensics, no prod impact, no re-execution. **This is the workflow you'll use weekly.** ### Vendor evaluation without a parallel-instrumentation week You're on Phoenix; you want to evaluate Braintrust before switching. Without replay you'd instrument your agent to dual-write for a week, pay both, wait, decide. With replay: pipe last week's archived JSONL into Braintrust in 30 seconds. Decision before lunch. ### Backfill a sink you just added Six months of archived traces; today you sign up for OpenPipe to fine-tune. Pipe the archive through an OpenPipe sink — six months of training data backfilled in one command, not from-now-forward only. ### Smoke-test a new routing rule About to add `{ match: { 'llm.cost.total': '>0.5' }, to: ['cost-alerts'] }`. Will it spam? Replay last week through it with a memory sink. See the actual volume before deploying. ### Why this is unique - Phoenix/Braintrust/etc. each own their data silo — you can't pipe Phoenix's stored traces into Braintrust without writing per-vendor ETL each time. - OTel Collector is stateless and push-only; no concept of replay. - Most tracing tools assume "live or never." `agent-otel` separates the transport format (OTel) from the routing decisions (rules). You can re-decide destinations indefinitely. ## Counterfactual replay — re-run a stored LLM call with one thing swapped `agent-otel/replay-execute` does what eval-platform playgrounds do, but **programmatically across many traces**. Take a stored LLM span, swap one thing (model, system prompt, temperature), call the real provider, get a real response. Not data-mutation — actual re-execution. ```ts import { replayLLMCall, swapModel, swapSystem, pipe } from 'agent-otel/replay-execute'; import { postgres } from 'agent-otel/sinks'; const archive = postgres({ url: process.env.DATABASE_URL! }); // "Would my agent have made a different decision with Sonnet 4.7?" const result = await replayLLMCall({ source: archive, spanId: '0123abcd...', // a stored LLM span mutate: swapModel('claude-sonnet-4-7'), provider: 'anthropic', apiKey: process.env.ANTHROPIC_API_KEY!, }); console.log('Original output:', result.originalSpan.attributes['llm.output_messages.0.message.content']); console.log('New output: ', result.newResponse.content); console.log('Cost: ', result.newResponse.tokens); ``` Composable mutators: `swapModel`, `swapSystem`, `setTemperature`, `appendMessage`, plus `pipe(...)` to chain them. Bring your own with `(req) => mutated`. Works with any provider via the `execute` callback: ```ts import OpenAI from 'openai'; const openai = new OpenAI({ apiKey: process.env.OPENAI_API_KEY! }); await replayLLMCall({ source: archive, spanId, mutate: swapModel('gpt-5'), execute: async (req) => { const resp = await openai.chat.completions.create(req as any); return { content: resp.choices[0].message.content ?? undefined, raw: resp }; }, }); ``` Built-in `provider: 'anthropic'` lazy-loads `@anthropic-ai/sdk` (optional peer dep). For OpenAI/Gemini/etc. supply your own `execute` until first-class adapters ship. `dryRun: true` returns the mutated request without calling the provider — useful for "what does the request look like with my mutator applied" before paying for tokens. **Concrete workflow this enables — replay → eval pipeline:** 1. Pull yesterday's failed traces from postgres (`scry query --status=ERROR --since=24h`) 2. For each, `replayLLMCall` with `swapModel('claude-sonnet-4-7')` 3. Pipe new responses into a Braintrust experiment for scoring 4. Decision: did upgrading the model fix more than it broke? Their playground × N, scripted, repeatable. E2E tested against the real Anthropic API in `tests/e2e/replay-execute.test.ts`. > **Note.** This is the **counterfactual single-LLM-call** flavor — re-runs ONE node of the trace. Re-executing the entire downstream subtree (so a tool's new response cascades) is a bigger feature on the roadmap; the single-call version covers the most common "what if I'd used the new model" workflow today. ## Sinks shipped today | Sink | Module | What it does | |---|---|---| | Phoenix | `agent-otel/sinks/phoenix` | OTLP/HTTP to Phoenix. Self-hosted or cloud. Optional API key. | | Braintrust | `agent-otel/sinks/braintrust` | OTLP/HTTP to Braintrust. Routes to a project's logs or an experiment. | | Slack | `agent-otel/sinks/slack` | Posts spans as messages to a Slack incoming webhook. Built-in rate limiting. Pretty default formatter; bring your own. | | Generic OTLP | `agent-otel/sinks/otlp` | Any OTLP/HTTP endpoint. Works with Honeycomb, Datadog, Tempo, Jaeger v2, LangSmith, Langfuse, anything that speaks OTLP. Defaults to protobuf via the official OTel exporter; JSON available as fallback. | | S3 (and S3-compatible) | `agent-otel/sinks/s3` | Gzipped JSONL upload to S3 / R2 / MinIO / Backblaze. The cheap canonical archive sink. `@aws-sdk/client-s3` is an optional peer dep — install only if you use this sink. | | Postgres | `agent-otel/sinks/postgres` | Insert spans into a Postgres table. Default OTel-canonical schema (or BYO via `columnMapper`). `ON CONFLICT (span_id) DO UPDATE` with JSONB attribute merge. Also the backing store `scry` queries. `postgres` is an optional peer dep — only required when using `url`. | | In-memory | `agent-otel/sinks/memory` | JS array. Tests and replay. | | JSONL file | `agent-otel/sinks/jsonl` | Append per span to a local file. Single-process. | Planned: Sentry, OpenPipe, console pretty-printer, GCS native (vs S3-compat), generic webhook helper. ## Privacy: vendors see fakes, you keep the real `agent-otel/privacy` wraps any sink so spans are PII-masked before consumption. Powered by [`pii-proxy`](https://github.com/mirkokiefer/pii-proxy) — replaces real PII with plausible fakes (not tokens, so LLM reasoning quality is preserved) via a bijective map. Real values stay in your canonical archive; vendors only ever see the fakes; round-tripping LLM responses still works because the map unmasks them back. ```ts import { withPrivacy, PrivacyProxy } from 'agent-otel/privacy'; const proxy = new PrivacyProxy(); // shared across wrapped sinks → consistent fakes const router = defineRouter({ sinks: { archive: jsonl({ path: './canonical.jsonl' }), // RAW phoenix: withPrivacy(phoenix({ ... }), { proxy }), // MASKED braintrust: withPrivacy(braintrust({ ... }), { proxy }), // MASKED — same fakes as Phoenix }, rules: [{ match: '*', to: ['archive', 'phoenix', 'braintrust'] }], }); ``` Output (real run, verified by e2e against the live Braintrust API in `tests/e2e/privacy-braintrust.test.ts`): ``` ARCHIVE → "user.email": "mirko-test-abcd@kiefer.com", "auth.token": "sk-secret-..." BRAINTRUST → "user.email": "herman21@yahoo.com", "auth.token": "[redacted]" PHOENIX → "user.email": "herman21@yahoo.com", "auth.token": "[redacted]" ← same fakes ``` Knobs: - `redactKeys` — hard-redact specific attribute keys (auth tokens, secrets) instead of masking — replaced with literal `'[redacted]'` - `passthroughKeys` — skip masking for non-PII keys that pii-proxy might over-detect (e.g., span markers you need to find your event later) - `maskNames` — also mask span name + status_message (default: false) - Map is JSON-serializable via `exportProxyMap` / `importProxyMap` for cross-process persistence pii-proxy auto-detects: emails, phone numbers, IBAN/credit cards, IPs, named entities. Custom-format strings (tracking numbers, internal IDs) need either an explicit `redactKeys` entry or a custom detector — write one if your spans carry custom-shape PII. This composition is uniquely ours. Phoenix/Braintrust/Datadog don't offer it. The OTel Collector has destructive redaction processors only — non-reversible. Reversible privacy + multi-vendor routing has not existed until now. ## Filter grammar Match expressions for routing rules. Keys are OTel attribute paths or top-level fields (`kind`, `status_code`). ```ts { match: '*' } // every span { match: { kind: 'CLIENT' } } // top-level field { match: { status_code: 'ERROR' } } // top-level field { match: { 'gen_ai.system': '*' } } // attribute presence { match: { 'gen_ai.system': 'anthropic' } } // exact equality { match: { 'llm.cost.total': '>0.1' } } // numeric: >, <, >=, <= { match: { foo: '!=bar' } } // explicit inequality { match: { foo: '==bar' } } // explicit equality { match: { a: 'x', b: 'y' } } // multiple keys → AND { match: [{ a: 'x' }, { b: 'y' }] } // array → OR ``` Multiple **rules** matching the same span union their target sinks. ## Design principles 1. **OTel-canonical input.** You emit standard OTel spans. The router is just a `SpanProcessor`. No new SDK to learn. 2. **Sink adapters, not lock-in.** Every sink translates OTel spans to that sink's format internally. Change a sink, app code unchanged. 3. **Attribute-based routing.** Rules match on span attributes — agent semantic conventions are first-class. 4. **No required storage.** The router is streaming. Want durability? Plug in a storage sink. Many setups use multiple. 5. **Bring your own backends.** Built-in sinks are reference implementations. Anyone can write a new sink in ~50 lines. ## Status **v0.0.16 — pre-alpha.** Core router, eight reference sinks (memory/jsonl/otlp/phoenix/braintrust/slack/s3/postgres), replay (re-route flavor) + replay-execute (counterfactual single-LLM-call flavor), reversible PII masking via `agent-otel/privacy`, **auto-instrument for `@anthropic-ai/sdk` and `openai`** via `agent-otel/anthropic` + `agent-otel/openai`, `scry` CLI with query/trace/chain/stats subcommands (local-DB + remote-endpoint modes). 102 unit tests + 8 e2e tests against real backends — including end-to-end verified `withPrivacy(braintrust())` (POST → fetch back, real values masked, fakes present), `replayLLMCall` against real Anthropic, and `instrument(...)` emitting OpenInference spans against both Anthropic and OpenAI APIs. API will change. Open issues, send PRs. ## Tests ```bash bun test # unit tests (fast, no network) bun run test:e2e # end-to-end tests against real backends # (skips per-test if env vars not set) ``` E2E tests verify each sink against a real backend. Required env vars and what's tested are documented in `tests/e2e/README.md`. CI without secrets passes (skip-if-missing pattern); local runs verify whatever you have keys for. The package is independent — no required hosted account, no preferred backend. Use it with whatever stack. ## What's next - **MCP HTTP / SSE transport** — `scry mcp --transport=http --port=N` for org-internal multi-user setups. Stdio shipped in v0.0.16; HTTP is incremental from there. - **More auto-instrument adapters** — `agent-otel/vercel-ai`, `agent-otel/mastra`, `agent-otel/openai-responses` (the new Responses API). Anthropic + OpenAI Chat Completions shipped. Streaming wrap for both lands next. - **Subtree re-execution** — extend `replayLLMCall` to re-run downstream tools/LLMs from the swap point, not just one node. Bridges to RL rollouts. - **Healthcare/PHI detector preset** — `withPrivacy(sink, { preset: 'hipaa' })` bundling ICD-10 / NPI / MRN detectors on top of pii-proxy. - **Annotation write-back** — agents record observations on past spans (their own labels for self-supervised eval data). ## License MIT.