Content
# `agent-otel` + `scry`
> **Agent-native observability, in two layers.**
> `agent-otel` — the OTel-native router + sinks + replay (the substrate).
> `scry` — the SDK and CLI an agent uses to query its own traces.
🚧 v0.0.20 — pre-alpha, APIs may change. MIT.
---
`agent-otel` is the substrate: declarative fanout to any number of backends, replay for retroactive rerouting, reversible PII masking. App engineers wire it up the same way whether the consumer is a human, an agent, or both. `scry` is where the agent-first thesis lives: an SDK and CLI for an agent to inspect its own traces in-process or from a shell. Think `kubernetes` + `kubectl` — library and CLI, dual-named on purpose. Phoenix/Braintrust/Langfuse render traces for humans; `scry` gives an agent a query surface over the same data. They compose.
## Install
```bash
npm install agent-otel
# or: bun add agent-otel
```
`scry` ships as a CLI in the same package:
```bash
npx scry --help
# after bun add agent-otel:
bunx scry --help
```
## 60-second install — Anthropic / OpenAI + Braintrust
If you're on `@anthropic-ai/sdk` or `openai` and Braintrust today, this is the whole setup:
```ts
import Anthropic from '@anthropic-ai/sdk';
import { NodeSDK } from '@opentelemetry/sdk-node';
import { defineRouter } from 'agent-otel';
import { instrument } from 'agent-otel/anthropic';
import { braintrust, postgres } from 'agent-otel/sinks';
import { withPrivacy, PrivacyProxy } from 'agent-otel/privacy';
// 1. Wire the router → backends
const proxy = new PrivacyProxy();
const router = defineRouter({
sinks: {
braintrust: withPrivacy(
braintrust({ apiKey: process.env.BRAINTRUST_API_KEY!, project: 'support-agent' }),
{ proxy, redactKeys: ['auth.token'] }, // PII never reaches Braintrust
),
archive: postgres({ url: process.env.DATABASE_URL! }), // your own escape hatch
},
rules: [{ match: '*', to: ['braintrust', 'archive'] }],
});
new NodeSDK({ spanProcessors: [router.asSpanProcessor()] }).start();
// 2. Wrap your client. That's it. Every call now emits a perfect
// OpenInference span — to Braintrust (masked) AND your archive (raw).
const anthropic = instrument(new Anthropic({ apiKey: process.env.ANTHROPIC_API_KEY! }));
const resp = await anthropic.messages.create({
model: 'claude-sonnet-4-7',
max_tokens: 1024,
messages: [{ role: 'user', content: 'Hello' }],
});
```
**For OpenAI, swap two lines:**
```ts
import OpenAI from 'openai';
import { instrument as instrumentOpenAI } from 'agent-otel/openai';
const openai = instrumentOpenAI(new OpenAI({ apiKey: process.env.OPENAI_API_KEY! }));
const resp = await openai.chat.completions.create({
model: 'gpt-5.5',
messages: [{ role: 'user', content: 'Hello' }],
});
```
Same auto-instrumentation. Built-in cost tables for GPT-5.5 / GPT-5 / GPT-4.1 / GPT-4o families; date-pinned model IDs (`gpt-5.5-2026-04-23`) match by prefix.
What you get for those ~10 lines:
- ✅ Every Anthropic / OpenAI call traced with OpenInference attributes (gen_ai.\*, llm.\*, tool calls flattened)
- ✅ Braintrust dashboards work as before (real evals, real playground), **but with PII masked**
- ✅ Your own Postgres archive — query with `scry trace tree <id>` from the CLI
- ✅ Real production cost and token counts on every span (Sonnet/Opus/Haiku tables built in)
- ✅ Replay any stored call counterfactually (`replayLLMCall`) without re-running your whole agent
E2E tested against the real Anthropic + Braintrust APIs (`tests/e2e/instrument-anthropic.test.ts`, `tests/e2e/privacy-braintrust.test.ts`).
For OpenAI, Vercel AI SDK, Mastra, CrewAI — auto-instrument adapters are next. Today: use OpenInference's per-vendor packages alongside agent-otel's router (they emit OTel; we route OTel; same wire format).
## Router — one OTel emit, declarative fanout
```ts
import { NodeSDK } from '@opentelemetry/sdk-node';
import { defineRouter } from 'agent-otel';
import { phoenix, braintrust, slack, jsonl } from 'agent-otel/sinks';
const router = defineRouter({
sinks: {
phoenix: phoenix({ endpoint: process.env.PHOENIX_ENDPOINT, apiKey: process.env.PHOENIX_API_KEY }),
braintrust: braintrust({ apiKey: process.env.BRAINTRUST_API_KEY!, project: 'support-agent' }),
alerts: slack({ webhookUrl: process.env.SLACK_WEBHOOK_URL! }),
archive: jsonl({ path: './traces.jsonl' }),
},
rules: [
// Everything → archival
{ match: '*', to: ['archive'] },
// LLM calls → both eval platforms
{ match: { 'gen_ai.system': '*' }, to: ['phoenix', 'braintrust'] },
// Expensive LLM calls → ping #ai-cost-watch in Slack
{ match: { 'llm.cost.total': '>1.0' }, to: ['alerts'] },
// Errors → Slack AND Braintrust (so eval picks them up)
{ match: { 'status_code': 'ERROR' }, to: ['alerts', 'braintrust'] },
],
});
const sdk = new NodeSDK({ spanProcessors: [router.asSpanProcessor()] });
sdk.start();
```
Your existing `tracer.startSpan(...)` calls now fan out per the rules. Add a sink, drop a sink, change a threshold — config-only, no app-code changes.
## `scry` — SDK and CLI for agents to query their own traces
Two paths depending on where your agent runs. Pick one or use both:
| Where the agent runs | Reach for | Looks like |
|---|---|---|
| **In-process** (Node.js / Bun, Vercel AI SDK, Anthropic SDK directly) | The **TypeScript SDK** | `sink.findSpans({ status_code: 'ERROR' })` from inside the same process the agent runs in |
| **In a sandbox shell** (E2B, Daytona, your own Docker, anywhere with bash) | The **`scry` CLI** | `scry query --status=ERROR | jq` piped through standard shell tools |
Both surfaces operate on the same data (an `Inspectable` sink — `memory` and `postgres` today, others can implement it). Pick whichever fits where the agent actually lives.
### Programmatic (in-process SDK — most agent code)
For agents running in your Node/Bun process. Import the primitives directly; no shell, no JSON round-trip. Use this when your agent is a function in a TypeScript codebase calling LLMs/tools.
```ts
import { memory } from 'agent-otel/sinks/memory';
import { and, substring } from 'agent-otel/filters';
import { buildTree, causalChain, renderTree } from 'agent-otel/trace-tree';
const sink = memory();
// ... router emits into sink ...
const errors = sink.findSpans(
and({ status_code: 'ERROR' }, substring('name', 'tool.')),
{ limit: 20 },
);
const tree = buildTree(sink.getTrace(traceId));
console.log(renderTree(tree, { attrs: ['llm.cost.total'] }));
```
### MCP server (any MCP-aware agent: Claude Code, Cursor, Devin, …)
Run `scry mcp` and any MCP client gets a tool surface for trace inspection. Local-dev pattern is to wire it into your client config so it spawns as a subprocess on demand:
```json
// .claude/settings.json (Claude Code) — Cursor / Devin / etc. take similar config
{
"mcpServers": {
"scry": {
"command": "npx",
"args": ["scry", "mcp", "--db", "postgres://localhost/myapp"]
}
}
}
```
Or against a remote scry HTTP endpoint (in-sandbox / org-wide setups where the JWT was minted server-side):
```json
{
"mcpServers": {
"scry": {
"command": "npx",
"args": ["scry", "mcp", "--endpoint", "https://api.example.com/v1/scry", "--token", "$SCRY_TOKEN"]
}
}
}
```
Tools registered:
- **`scry_query_jobs`** — list recent agent jobs (filter by status / attribute)
- **`scry_get_trace`** — render a trace as an ASCII tree by `trace_id`
- **`scry_causal_chain`** — walk root → target span path
- **`scry_stats`** — aggregate counts / cost / duration / errors
The same primitives that power the CLI and the SDK, exposed over MCP. First MCP server in LLM-trace-land.
### CLI (sandbox shell — and dev terminals)
For agents running in a sandbox shell (E2B, Daytona, etc.) AND for human engineers debugging from a laptop. Composes with shell tools naturally.
**Connect via direct DB or remote endpoint:**
```bash
export SCRY_DB=postgres://localhost/mydb # direct Postgres (local / dev)
# or for remote:
export SCRY_ENDPOINT=https://scry.example.com
export SCRY_TOKEN=<jwt>
```
Flags `--db`, `--endpoint`, `--token` work per-call too.
**Three one-liners:**
```bash
# Find all ERROR spans in the last 10 minutes, extract span IDs
scry query --status=ERROR --since=10m --output=json | jq '.[] | .spanId'
# Render the full call tree of a job (LLM ↔ tool ↔ DB) as ASCII
scry trace tree 0123abcd...
# Aggregate cost, latency, error rate across a filter
scry stats --attr=gen_ai.system=anthropic
```
Full subcommand reference:
```
scry query [--status=X] [--kind=X] [--name=X] [--attr=k=v] [--since=10m] [--limit=N]
scry trace get <trace_id>
scry trace tree <trace_id> [--attrs=k1,k2]
scry chain <trace_id> <span_id> # walk a span back to root: what led to this error?
scry stats [--status=X] [--attr=k=v]
```
Composes naturally with shell tooling: `scry query --output=json | jq`, `scry stats | awk '$1 > 0.1 {exit 1}'`. No MCP boot, no ceremony.
---
## Cost tracking — `agent-otel/cost`
Standard contract + OTel-native attributes for LLM cost. Three primitives:
```ts
import {
calculateCost,
recordLLMCall,
extractors,
type PricingSource,
} from 'agent-otel/cost';
// You supply pricing — agent-otel ships none. (Tables go stale; we don't
// want to be the maintainer.) See examples/pricing-static.ts.
const myPricing: PricingSource = { lookup: (m) => MY_TABLE[m] };
// In your provider stream's finish handler:
const usage = extractors.openai(chunk.usage); // → cross-provider LLMUsage
const cost = calculateCost('gpt-5', usage, myPricing); // → { cost, costType, breakdown }
recordLLMCall(span, { usage, cost }); // → dual-write attrs
```
The span now carries **both** OpenInference (`llm.cost.total`, `llm.token_count.*`) **and** OTel-GenAI (`gen_ai.cost.total`, `gen_ai.usage.input_tokens`) — Phoenix, Arize, Langfuse, scry, Datadog GenAI, and any future OTel-GenAI consumer read it identically.
### Why "bring your own pricing"?
Pricing data goes stale weekly (new models drop, providers re-price prompt cache). Baking a table into agent-otel would mean a release every time. We ship the **contract** — you wire a static map, fetch from [models.dev](https://models.dev), pull LiteLLM's JSON, hit your billing DB, whatever. See `examples/pricing-*.ts`.
### What you get on the span
| Attribute | Both writes (OpenInference + OTel-GenAI) |
|---|---|
| Uncached input tokens | `llm.token_count.prompt`, `gen_ai.usage.input_tokens` |
| Output tokens | `llm.token_count.completion`, `gen_ai.usage.output_tokens` |
| Cache read / write | `llm.token_count.prompt_details.cache_read/write` |
| Reasoning tokens | `llm.token_count.completion_details.reasoning` |
| Total USD cost | `llm.cost.total`, `gen_ai.cost.total` |
| Per-bucket breakdown | `gen_ai.cost.input/output/cache_read/cache_write/reasoning` |
| Provenance | `gen_ai.cost.type` — `'actual' \| 'estimated' \| 'unknown'` |
Provider-reported actuals (OpenRouter exposes `usage.cost`) are preferred over table-based estimates — set on `LLMUsage.provider_cost` by the OpenAI extractor when present.
### Provider extractors
Pure functions, raw provider chunk → `LLMUsage`. Each knows the provider's quirks (OpenAI subtracts `cached_tokens` from `prompt_tokens`; Anthropic's `input_tokens` is already uncached; Gemini uses `cachedContentTokenCount`).
```ts
extractors.openai(chunk.usage) // OpenAI / OpenRouter / Azure OpenAI
extractors.anthropic(message.usage) // Claude / Bedrock Anthropic
extractors.gemini(response.usageMetadata)
```
## What this replaces
You're probably doing some of these by hand right now:
- **Phoenix SDK** for traces. **Braintrust SDK** for evals. **Datadog OTLP** for APM. **Sentry SDK** for errors.
- A custom Slack/Discord script that scrapes logs for "LLM call > $X" alerts.
- A nightly export script that copies a sample of traces to JSONL for fine-tuning later.
- An ad-hoc adapter that reformats traces when you change eval vendors.
`agent-otel` collapses all of that into one OTel emit + a declarative routing config. Same wire format everywhere; backends are just sinks.
## `agent-otel` × [OpenInference](https://github.com/Arize-ai/openinference)
Different layers in the same pipeline:
| | OpenInference | `agent-otel` |
|---|---|---|
| **Lives at** | SDK boundary (input) — wraps the LLM SDK so calls emit spans | Export boundary (output) — routes spans to sinks |
| **Wraps** | Specific LLM SDKs | Nothing — consumes any OTel emitter |
| **Replay / cost-aware sampling** | No | Yes |
Use both: OpenInference makes your Anthropic SDK calls emit a span; `agent-otel` decides that span goes to Phoenix + Slack but not Braintrust. Convention compatibility is covered above.
## Convention modes — OpenInference, OTel GenAI, or both
`agent-otel`'s instrumentations emit LLM telemetry under two parallel attribute name sets:
- **OpenInference** (`llm.*`, `openinference.span.kind`, flattened messages) — production-mature, what Phoenix renders natively.
- **OTel GenAI semantic conventions** (`gen_ai.*`, `gen_ai.operation.name`, structured messages) — the official direction; client spans went stable in early 2026, agent/tool spans still Development.
Three modes, env-var-controlled — matches OpenTelemetry's [`OTEL_SEMCONV_STABILITY_OPT_IN`](https://opentelemetry.io/docs/specs/semconv/general/attribute-naming/#opt-in-attributes) pattern used for the HTTP semconv migration. Reusing the OTel-canonical env var means ops folks recognize it; no Daslab-specific flags.
| Mode | Env var | What gets emitted | When to use |
|---|---|---|---|
| `openinference` | `OTEL_SEMCONV_STABILITY_OPT_IN=openinference` | OpenInference only (`llm.*`, flat messages) | Phoenix-only stack, no OTel-GenAI-native consumer |
| `dup` (default) | unset or `gen_ai_dup` | Scalars in both name sets; content stays flat OpenInference | **Recommended.** Mixed environments. Phoenix still renders, OTel-native backends pick up `gen_ai.*` for free. |
| `gen_ai` | `OTEL_SEMCONV_STABILITY_OPT_IN=gen_ai_latest_experimental` | `gen_ai.*` scalars only; content attributes skipped until structured `gen_ai.input.messages` JSON lands in a follow-up | Greenfield, OTel-native backends, willing to skip content until structured emission lands |
Programmatic override for per-client control:
```ts
import { instrument } from 'agent-otel/anthropic';
const client = instrument(new Anthropic({...}), { conventionMode: 'dup' });
```
### Why `dup` is the default
Scalar dual-emit adds ~1-2 KB per span — negligible against the 64 KB raw-request blob already emitted. In exchange, the same span renders cleanly in Phoenix *and* shows up as a GenAI span in Honeycomb / Langfuse / DataDog with no flag set. The library serves both audiences out of the box.
### Content attribute pollution — why content stays flat
The cheap part of dual-emit is scalars (model, tokens, finish reasons — ~10 attributes, ~1-2 KB). The expensive part is content: re-emitting full message bodies as `gen_ai.input.messages` JSON while still keeping flattened `llm.input_messages.N.*` would double per-span payload for message content. OTel's own guidance is to gate content attributes on opt-in; we currently keep content flat-OpenInference across `openinference` and `dup` modes. Structured `gen_ai.input.messages` JSON for `gen_ai` mode lands in a follow-up — for now, `gen_ai` mode emits scalars only and content attributes are skipped.
### Deprecation cadence
When OTel GenAI agent/tool/eval spans are stable and Phoenix natively renders OTel GenAI attributes well, the default flips from `dup` → `gen_ai` (major-version bump). OpenInference remains supported via `OTEL_SEMCONV_STABILITY_OPT_IN=openinference` for two more minor releases, then dropped. Expected window: 6-12 months. Until then, `dup` carries no breaking changes.
## `agent-otel` × [OTel Collector](https://opentelemetry.io/docs/collector/)
Both route OTel data; different runtimes and different audiences.
- **Collector** is a Go sidecar configured in YAML — canonical for traditional APM with 100+ exporters in contrib.
- **agent-otel** is a TypeScript library you `npm install` — agent-aware (knows `gen_ai.*`, `llm.cost.total`), ships sinks for eval/training platforms (Phoenix-as-dataset, Braintrust experiments, OpenPipe) the Collector doesn't have, and the wire format is the same.
Run them alongside. They don't compete.
## `agent-otel` × Braintrust / Phoenix / Langfuse / LangSmith
`agent-otel` sits in front of these — your eval/observability backend keeps its job; PII masking, a vendor-neutral archive, programmatic agent-side query, and replay layer on top.
For an existing Braintrust user (same pattern works for Phoenix / Langfuse / LangSmith):
```ts
import { defineRouter } from 'agent-otel';
import { jsonl, postgres, braintrust } from 'agent-otel/sinks';
import { withPrivacy, PrivacyProxy } from 'agent-otel/privacy';
const proxy = new PrivacyProxy();
const router = defineRouter({
sinks: {
// KEEP: Braintrust as your eval/playground/experiments backend.
// ADD: PII masking so customer emails / tokens never reach Braintrust.
braintrust: withPrivacy(
braintrust({ apiKey: process.env.BRAINTRUST_API_KEY!, project: 'support-agent' }),
{ proxy, redactKeys: ['auth.token'] },
),
// ADD: vendor-neutral local archive — escape hatch + audit trail
archive: postgres({ url: process.env.DATABASE_URL!, table: 'spans' }),
// ADD: cheap on-disk dump for backfill / replay later
jsonl: jsonl({ path: './prod-traces.jsonl' }),
},
rules: [{ match: '*', to: ['braintrust', 'archive', 'jsonl'] }],
});
```
What this adds on top of Braintrust:
| Need | Braintrust alone | + agent-otel |
|---|---|---|
| Eval / playground / experiments | ✓ | ✓ (unchanged) |
| Trace ingest + dashboards | ✓ | ✓ (unchanged) |
| **PII masking before vendor sees it** | ✗ | ✓ via `withPrivacy()` (e2e tested against live Braintrust API) |
| **Vendor-neutral archive** (Postgres / S3 / JSONL) | ✗ | ✓ |
| **Programmatic agent self-debug** (`scry` SDK + CLI) | ✗ | ✓ |
| **Counterfactual replay** ("what if Sonnet 4.7?") | manual playground only | `replayLLMCall()` — see below |
| **MCP server** for Claude Code / Cursor to query traces | ✗ | planned |
| **Lock-in escape** — leave whenever | hard | trivial; spans archived in your own store |
Keep what works, add what's missing.
## Replay — retroactive routing
The unique capability `agent-otel` unlocks: **change your mind about where spans go AFTER you've collected them.** Routing is configuration, not code, so the destinations aren't baked in at emit time.
```ts
import { replay, fromJsonl } from 'agent-otel/replay';
await replay({
source: fromJsonl('./prod-traces.jsonl'),
router: defineRouter({
sinks: { braintrust: braintrust({...}) },
rules: [{ match: '*', to: ['braintrust'] }],
}),
});
```
Take spans you already captured, re-route them through any router config. Concrete workflows this enables:
### Customer debugging without touching prod (the daily-driver use case)
A customer pings you: *"my agent broke yesterday at 3:14pm."*
```ts
await replay({
source: fromJsonl('./prod.jsonl'),
where: s => s.traceId === 'trace_xyz',
router: defineRouter({
sinks: { slack: slack({ webhookUrl: DEBUG_CHANNEL }) },
rules: [{ match: '*', to: ['slack'] }],
}),
});
```
Every step of that one trace pings you in Slack with attributes pretty-printed. Pure forensics, no prod impact, no re-execution. **This is the workflow you'll use weekly.**
### Vendor evaluation without a parallel-instrumentation week
You're on Phoenix; you want to evaluate Braintrust before switching. Without replay you'd instrument your agent to dual-write for a week, pay both, wait, decide. With replay: pipe last week's archived JSONL into Braintrust in 30 seconds. Decision before lunch.
### Backfill a sink you just added
Six months of archived traces; today you sign up for OpenPipe to fine-tune. Pipe the archive through an OpenPipe sink — six months of training data backfilled in one command, not from-now-forward only.
### Smoke-test a new routing rule
About to add `{ match: { 'llm.cost.total': '>0.5' }, to: ['cost-alerts'] }`. Will it spam? Replay last week through it with a memory sink. See the actual volume before deploying.
### Why this is unique
- Phoenix/Braintrust/etc. each own their data silo — you can't pipe Phoenix's stored traces into Braintrust without writing per-vendor ETL each time.
- OTel Collector is stateless and push-only; no concept of replay.
- Most tracing tools assume "live or never."
`agent-otel` separates the transport format (OTel) from the routing decisions (rules). You can re-decide destinations indefinitely.
## Counterfactual replay — re-run a stored LLM call with one thing swapped
`agent-otel/replay-execute` does what eval-platform playgrounds do, but **programmatically across many traces**. Take a stored LLM span, swap one thing (model, system prompt, temperature), call the real provider, get a real response. Not data-mutation — actual re-execution.
```ts
import { replayLLMCall, swapModel, swapSystem, pipe } from 'agent-otel/replay-execute';
import { postgres } from 'agent-otel/sinks';
const archive = postgres({ url: process.env.DATABASE_URL! });
// "Would my agent have made a different decision with Sonnet 4.7?"
const result = await replayLLMCall({
source: archive,
spanId: '0123abcd...', // a stored LLM span
mutate: swapModel('claude-sonnet-4-7'),
provider: 'anthropic',
apiKey: process.env.ANTHROPIC_API_KEY!,
});
console.log('Original output:', result.originalSpan.attributes['llm.output_messages.0.message.content']);
console.log('New output: ', result.newResponse.content);
console.log('Cost: ', result.newResponse.tokens);
```
Composable mutators: `swapModel`, `swapSystem`, `setTemperature`, `appendMessage`, plus `pipe(...)` to chain them. Bring your own with `(req) => mutated`.
Works with any provider via the `execute` callback:
```ts
import OpenAI from 'openai';
const openai = new OpenAI({ apiKey: process.env.OPENAI_API_KEY! });
await replayLLMCall({
source: archive,
spanId,
mutate: swapModel('gpt-5'),
execute: async (req) => {
const resp = await openai.chat.completions.create(req as any);
return { content: resp.choices[0].message.content ?? undefined, raw: resp };
},
});
```
Built-in `provider: 'anthropic'` lazy-loads `@anthropic-ai/sdk` (optional peer dep). For OpenAI/Gemini/etc. supply your own `execute` until first-class adapters ship.
`dryRun: true` returns the mutated request without calling the provider — useful for "what does the request look like with my mutator applied" before paying for tokens.
**Concrete workflow this enables — replay → eval pipeline:**
1. Pull yesterday's failed traces from postgres (`scry query --status=ERROR --since=24h`)
2. For each, `replayLLMCall` with `swapModel('claude-sonnet-4-7')`
3. Pipe new responses into a Braintrust experiment for scoring
4. Decision: did upgrading the model fix more than it broke?
Their playground × N, scripted, repeatable. E2E tested against the real Anthropic API in `tests/e2e/replay-execute.test.ts`.
> **Note.** This is the **counterfactual single-LLM-call** flavor — re-runs ONE node of the trace. Re-executing the entire downstream subtree (so a tool's new response cascades) is a bigger feature on the roadmap; the single-call version covers the most common "what if I'd used the new model" workflow today.
## Sinks shipped today
| Sink | Module | What it does |
|---|---|---|
| Phoenix | `agent-otel/sinks/phoenix` | OTLP/HTTP to Phoenix. Self-hosted or cloud. Optional API key. |
| Braintrust | `agent-otel/sinks/braintrust` | OTLP/HTTP to Braintrust. Routes to a project's logs or an experiment. |
| Slack | `agent-otel/sinks/slack` | Posts spans as messages to a Slack incoming webhook. Built-in rate limiting. Pretty default formatter; bring your own. |
| Generic OTLP | `agent-otel/sinks/otlp` | Any OTLP/HTTP endpoint. Works with Honeycomb, Datadog, Tempo, Jaeger v2, LangSmith, Langfuse, anything that speaks OTLP. Defaults to protobuf via the official OTel exporter; JSON available as fallback. |
| S3 (and S3-compatible) | `agent-otel/sinks/s3` | Gzipped JSONL upload to S3 / R2 / MinIO / Backblaze. The cheap canonical archive sink. `@aws-sdk/client-s3` is an optional peer dep — install only if you use this sink. |
| Postgres | `agent-otel/sinks/postgres` | Insert spans into a Postgres table. Default OTel-canonical schema (or BYO via `columnMapper`). `ON CONFLICT (span_id) DO UPDATE` with JSONB attribute merge. Also the backing store `scry` queries. `postgres` is an optional peer dep — only required when using `url`. |
| In-memory | `agent-otel/sinks/memory` | JS array. Tests and replay. |
| JSONL file | `agent-otel/sinks/jsonl` | Append per span to a local file. Single-process. |
Planned: Sentry, OpenPipe, console pretty-printer, GCS native (vs S3-compat), generic webhook helper.
## Privacy: vendors see fakes, you keep the real
`agent-otel/privacy` wraps any sink so spans are PII-masked before consumption. Powered by [`pii-proxy`](https://github.com/mirkokiefer/pii-proxy) — replaces real PII with plausible fakes (not tokens, so LLM reasoning quality is preserved) via a bijective map. Real values stay in your canonical archive; vendors only ever see the fakes; round-tripping LLM responses still works because the map unmasks them back.
```ts
import { withPrivacy, PrivacyProxy } from 'agent-otel/privacy';
const proxy = new PrivacyProxy(); // shared across wrapped sinks → consistent fakes
const router = defineRouter({
sinks: {
archive: jsonl({ path: './canonical.jsonl' }), // RAW
phoenix: withPrivacy(phoenix({ ... }), { proxy }), // MASKED
braintrust: withPrivacy(braintrust({ ... }), { proxy }), // MASKED — same fakes as Phoenix
},
rules: [{ match: '*', to: ['archive', 'phoenix', 'braintrust'] }],
});
```
Output (real run, verified by e2e against the live Braintrust API in `tests/e2e/privacy-braintrust.test.ts`):
```
ARCHIVE → "user.email": "mirko-test-abcd@kiefer.com", "auth.token": "sk-secret-..."
BRAINTRUST → "user.email": "herman21@yahoo.com", "auth.token": "[redacted]"
PHOENIX → "user.email": "herman21@yahoo.com", "auth.token": "[redacted]" ← same fakes
```
Knobs:
- `redactKeys` — hard-redact specific attribute keys (auth tokens, secrets) instead of masking — replaced with literal `'[redacted]'`
- `passthroughKeys` — skip masking for non-PII keys that pii-proxy might over-detect (e.g., span markers you need to find your event later)
- `maskNames` — also mask span name + status_message (default: false)
- Map is JSON-serializable via `exportProxyMap` / `importProxyMap` for cross-process persistence
pii-proxy auto-detects: emails, phone numbers, IBAN/credit cards, IPs, named entities. Custom-format strings (tracking numbers, internal IDs) need either an explicit `redactKeys` entry or a custom detector — write one if your spans carry custom-shape PII.
This composition is uniquely ours. Phoenix/Braintrust/Datadog don't offer it. The OTel Collector has destructive redaction processors only — non-reversible. Reversible privacy + multi-vendor routing has not existed until now.
## Filter grammar
Match expressions for routing rules. Keys are OTel attribute paths or top-level fields (`kind`, `status_code`).
```ts
{ match: '*' } // every span
{ match: { kind: 'CLIENT' } } // top-level field
{ match: { status_code: 'ERROR' } } // top-level field
{ match: { 'gen_ai.system': '*' } } // attribute presence
{ match: { 'gen_ai.system': 'anthropic' } } // exact equality
{ match: { 'llm.cost.total': '>0.1' } } // numeric: >, <, >=, <=
{ match: { foo: '!=bar' } } // explicit inequality
{ match: { foo: '==bar' } } // explicit equality
{ match: { a: 'x', b: 'y' } } // multiple keys → AND
{ match: [{ a: 'x' }, { b: 'y' }] } // array → OR
```
Multiple **rules** matching the same span union their target sinks.
## Design principles
1. **OTel-canonical input.** You emit standard OTel spans. The router is just a `SpanProcessor`. No new SDK to learn.
2. **Sink adapters, not lock-in.** Every sink translates OTel spans to that sink's format internally. Change a sink, app code unchanged.
3. **Attribute-based routing.** Rules match on span attributes — agent semantic conventions are first-class.
4. **No required storage.** The router is streaming. Want durability? Plug in a storage sink. Many setups use multiple.
5. **Bring your own backends.** Built-in sinks are reference implementations. Anyone can write a new sink in ~50 lines.
## Status
**v0.0.16 — pre-alpha.** Core router, eight reference sinks (memory/jsonl/otlp/phoenix/braintrust/slack/s3/postgres), replay (re-route flavor) + replay-execute (counterfactual single-LLM-call flavor), reversible PII masking via `agent-otel/privacy`, **auto-instrument for `@anthropic-ai/sdk` and `openai`** via `agent-otel/anthropic` + `agent-otel/openai`, `scry` CLI with query/trace/chain/stats subcommands (local-DB + remote-endpoint modes). 102 unit tests + 8 e2e tests against real backends — including end-to-end verified `withPrivacy(braintrust())` (POST → fetch back, real values masked, fakes present), `replayLLMCall` against real Anthropic, and `instrument(...)` emitting OpenInference spans against both Anthropic and OpenAI APIs. API will change. Open issues, send PRs.
## Tests
```bash
bun test # unit tests (fast, no network)
bun run test:e2e # end-to-end tests against real backends
# (skips per-test if env vars not set)
```
E2E tests verify each sink against a real backend. Required env vars and what's tested are documented in `tests/e2e/README.md`. CI without secrets passes (skip-if-missing pattern); local runs verify whatever you have keys for.
The package is independent — no required hosted account, no preferred backend. Use it with whatever stack.
## What's next
- **MCP HTTP / SSE transport** — `scry mcp --transport=http --port=N` for org-internal multi-user setups. Stdio shipped in v0.0.16; HTTP is incremental from there.
- **More auto-instrument adapters** — `agent-otel/vercel-ai`, `agent-otel/mastra`, `agent-otel/openai-responses` (the new Responses API). Anthropic + OpenAI Chat Completions shipped. Streaming wrap for both lands next.
- **Subtree re-execution** — extend `replayLLMCall` to re-run downstream tools/LLMs from the swap point, not just one node. Bridges to RL rollouts.
- **Healthcare/PHI detector preset** — `withPrivacy(sink, { preset: 'hipaa' })` bundling ICD-10 / NPI / MRN detectors on top of pii-proxy.
- **Annotation write-back** — agents record observations on past spans (their own labels for self-supervised eval data).
## License
MIT.
