Content
# Smart MCP Gateway
Smart MCP Gateway is a TypeScript/Node.js middleware between MCP clients and MCP servers.
It uses a small intent model to choose only the most relevant MCP tools, then forwards a pruned tool context to the final LLM.
This reduces token waste, improves routing precision, and gives better operational control than "send all MCP schemas every time".
Recommended runtime: Node.js `>=20.10.0`.
---
## Table Of Contents
1. [Project Goal](#project-goal)
2. [Core Features](#core-features)
3. [How Routing Works](#how-routing-works)
4. [Architecture](#architecture)
5. [Repository Structure](#repository-structure)
6. [Quick Start (Windows)](#quick-start-windows)
7. [Quick Start (Linuxmacos)](#quick-start-linuxmacos)
8. [Environment Variables](#environment-variables)
9. [Auth Header Matrix](#auth-header-matrix)
10. [API Usage](#api-usage)
11. [Evaluation And Quality Gates](#evaluation-and-quality-gates)
12. [Operations And Runtime](#operations-and-runtime)
13. [Troubleshooting](#troubleshooting)
14. [Security Checklist Before GitHub Publish](#security-checklist-before-github-publish)
15. [Known Limitations](#known-limitations)
---
## Project Goal
Traditional MCP usage often sends all available tools and schemas into every final-model call.
That is expensive and noisy.
This gateway changes the flow to:
1. User prompt enters gateway.
2. Small intent model distills capability needs.
3. Router scores and selects Top-K tools.
4. Gateway sends only selected tool schemas to final model.
5. Tool calls are proxied through unified policy and monitoring.
This is the main value proposition for this project.
---
## Core Features
- Intent distillation with multiple providers (`rule`, `openai`, `deepseek`, `qwen`, `moonshot`, `kimi`, `doubao`, `compatible`).
- Top-K routing with score thresholds, adaptive K, MMR diversity, and history boost.
- Session memory and long-term preference hints for multi-turn routing quality.
- Optional persistence of session memory across process restarts.
- Tool schema pruning for token-cost optimization.
- Gateway-level auth, admin auth, and monitoring auth.
- Runtime health probes and monitoring snapshots.
- Structured error responses and fallback behavior.
- Local evaluation tooling (token savings + semantic retrieval).
---
## How Routing Works
1. Query and context are distilled into capability requirements (`capabilities`, `keywords`, rewritten query).
2. Candidate tools are scored by lexical, semantic, and policy-aware signals.
3. Router applies:
- `ROUTER_SELECTION_MIN_SCORE`
- `ROUTER_SELECTION_MIN_RELATIVE_SCORE`
- optional adaptive K cutoff (`ROUTER_ADAPTIVE_K_*`)
- MMR reranking (`ROUTER_MMR_LAMBDA`)
- history boost (`ROUTER_HISTORY_*`)
4. Gateway returns:
- `selected_tools`
- `tool_context` (pruned schemas)
- `routing_explanations` (score + evidence)
- `token_saved_ratio`
---
## Architecture
```mermaid
flowchart LR
A[Client Prompt] --> B[/v1/gateway/chat]
B --> C[Intent Provider]
C --> D[Router and Top-K Selector]
D --> E[Pruned Tool Context]
E --> F[Final Response Provider]
F --> G[Gateway Response]
B --> H[/v1/gateway/tool-call]
H --> I[MCP Server]
J[/v1/admin/*] --> B
K[/v1/monitoring/*] --> B
```
---
## Repository Structure
```text
src/
api/ Express app, endpoints, auth middleware
intent/ Intent providers, session context, persistence
router/ Scoring, Top-K, MMR, history boost, explanations
registry/ MCP server/tool registry and persistence
metrics/ Runtime counters and quality metrics
errors/ Error model and API mapping
scripts/
smoke.ts
calc-token-savings.ts
eval-semantic-retrieval.ts
configs/
env.production.recommended
registry-state.json
router-vectors.sqlite
session-intent-memory.json
```
---
## Quick Start (Windows)
### 1) Install
```powershell
npm ci
```
### 2) Create local env
```powershell
Copy-Item .env.example .env
```
### 3) Minimum env
```env
PORT=8787
INTENT_PROVIDER=rule
CHAT_RESPONSE_PROVIDER=static
SEED_DEMO_DATA=true
MOCK_EXECUTOR=true
MCP_HTTP_TIMEOUT_MS=8000
ADMIN_TOKEN=change-me
```
### 4) Start service
```powershell
npm run start
```
### 5) Validate
```powershell
npm run build
npm test
npm run smoke
```
---
## Quick Start (LinuxmacOS)
### 1) Install
```bash
npm ci
```
### 2) Create local env
```bash
cp .env.example .env
```
### 3) Start service
```bash
npm run start
```
### 4) Validate
```bash
npm run build
npm test
npm run smoke
```
---
## Environment Variables
The full safe template is in `.env.example` and production suggestions are in `configs/env.production.recommended`.
### Core runtime
| Name | Default | Description |
|---|---|---|
| `PORT` | `8787` | HTTP listen port |
| `HTTP_JSON_BODY_LIMIT` | `1mb` | Max JSON body size, overflow returns `413` |
| `SEED_DEMO_DATA` | `true` | Seed demo registry on startup |
| `MOCK_EXECUTOR` | `false` | Allow local echo executor when tool endpoint missing |
### Provider selection
| Name | Default | Description |
|---|---|---|
| `INTENT_PROVIDER` | `rule` | Intent provider type |
| `INTENT_MODEL` | `gpt-4o-mini` | Intent model name |
| `INTENT_API_KEY` | unset | Generic key for compatible providers |
| `INTENT_BASE_URL` | unset | Base URL for compatible endpoint |
| `OPENAI_API_KEY` | unset | Required for OpenAI provider modes |
| `CHAT_RESPONSE_PROVIDER` | `static` | Final response provider (`static`, `openai`) |
| `CHAT_RESPONSE_MODEL` | `gpt-4o-mini` | Final response model |
### Timeout and retry
| Name | Default | Description |
|---|---|---|
| `INTENT_PROVIDER_TIMEOUT_MS` | `15000` | Intent request timeout |
| `INTENT_PROVIDER_MAX_RETRIES` | `1` | Intent retry count |
| `MCP_HTTP_TIMEOUT_MS` | `8000` | MCP tool-call timeout |
| `MCP_HTTP_MAX_RETRIES` | `1` | MCP tool-call retry count |
| `CHAT_RESPONSE_TIMEOUT_MS` | `15000` | Final response timeout |
| `CHAT_RESPONSE_MAX_RETRIES` | `1` | Final response retry count |
### Router quality controls
| Name | Default | Description |
|---|---|---|
| `ROUTER_SELECTION_MIN_SCORE` | `3` | Absolute score floor |
| `ROUTER_SELECTION_MIN_RELATIVE_SCORE` | `0.55` | Relative floor against top tool |
| `ROUTER_MMR_LAMBDA` | `0.8` | Relevance-diversity tradeoff (recommended `0.65-0.8`) |
| `ROUTER_ADAPTIVE_K_ENABLED` | `true` | Dynamic Top-K cutoff |
| `ROUTER_ADAPTIVE_K_DROP_THRESHOLD` | `0.4` | Adjacent drop trigger |
| `ROUTER_ADAPTIVE_K_MIN` | `1` | Minimum adaptive K |
| `ROUTER_HISTORY_BOOST_MAX` | `0.8` | Max history bias boost |
### Session memory and persistence
| Name | Default | Description |
|---|---|---|
| `SESSION_MEMORY_TTL_MS` | `1800000` | In-memory session TTL |
| `SESSION_MEMORY_MAX_RECENT_MESSAGES` | `8` | Messages kept per session |
| `SESSION_MEMORY_PERSIST_ENABLED` | `false` | Persist session memory across restart |
| `SESSION_MEMORY_PERSIST_PATH` | `configs/session-intent-memory.json` | Persistence file path |
| `SESSION_MEMORY_PERSIST_STRATEGY` | `on_shutdown` | `on_shutdown` or `on_turn` |
| `SESSION_MEMORY_PERSIST_MAX_TURNS` | `2000` | Max persisted turns |
### Auth and security
| Name | Default | Description |
|---|---|---|
| `ADMIN_TOKEN` | unset | Required for `/v1/admin/*` |
| `GATEWAY_AUTH_ENABLED` | `false` non-prod, `true` prod | Require auth for `/v1/gateway/*` |
| `GATEWAY_TOKEN` | unset | Gateway token (fallback to `ADMIN_TOKEN`) |
| `MONITORING_AUTH_ENABLED` | `true` | Require auth for `/v1/monitoring/*` |
| `MONITORING_TOKEN` | unset | Monitoring token (fallback to `ADMIN_TOKEN`) |
### Ops and monitoring
| Name | Default | Description |
|---|---|---|
| `OPS_READINESS_ENABLED` | `true` | Enable readiness gates |
| `OPS_READINESS_REQUIRE_TOOLS` | `false` | Require at least one registered tool |
| `OPS_RUNTIME_MONITORING_ENABLED` | `true` | Enable runtime monitoring endpoint |
| `OPS_REQUEST_LOGGING_ENABLED` | `true` | Structured request completion logs |
| `OPS_MAX_HEAP_USED_RATIO` | `0.92` | Heap ratio threshold for readiness |
---
## Auth Header Matrix
| Endpoint group | Required header | Notes |
|---|---|---|
| `/v1/gateway/*` | `x-gateway-token` | If missing, can fallback to `x-admin-token` |
| `/v1/admin/*` | `x-admin-token` | Always required |
| `/v1/monitoring/*` | `x-monitoring-token` | If missing, can fallback to `x-admin-token` |
If auth is enabled and token mismatch occurs, API returns `403`.
---
## API Usage
### 1) Health checks
Linux/macOS:
```bash
curl -X GET http://localhost:8787/health
curl -X GET http://localhost:8787/health/liveness
curl -X GET http://localhost:8787/health/readiness
```
Windows PowerShell:
```powershell
curl.exe -s http://localhost:8787/health
curl.exe -s http://localhost:8787/health/liveness
curl.exe -s http://localhost:8787/health/readiness
```
### 2) Gateway chat routing
Linux/macOS:
```bash
curl -X POST http://localhost:8787/v1/gateway/chat \
-H "content-type: application/json" \
-H "x-gateway-token: your-gateway-token" \
-d '{
"session_id":"sess-1",
"model":"gpt-4.1",
"client":"cursor",
"max_tools":3,
"messages":[{"role":"user","content":"Check failed CI workflow runs."}]
}'
```
Windows PowerShell:
```powershell
$body = @{
session_id = "sess-1"
model = "gpt-4.1"
client = "cursor"
max_tools = 3
messages = @(@{ role = "user"; content = "Check failed CI workflow runs." })
} | ConvertTo-Json -Depth 8
Invoke-RestMethod -Method Post `
-Uri "http://localhost:8787/v1/gateway/chat" `
-Headers @{ "x-gateway-token" = "your-gateway-token" } `
-ContentType "application/json" `
-Body $body
```
### 3) Tool call proxy
```bash
curl -X POST http://localhost:8787/v1/gateway/tool-call \
-H "content-type: application/json" \
-H "x-gateway-token: your-gateway-token" \
-d '{
"tool_name":"github.list_runs",
"arguments":{"owner":"acme","repo":"demo"}
}'
```
### 4) Monitoring runtime and quality
```bash
curl -X GET http://localhost:8787/v1/monitoring/runtime \
-H "x-monitoring-token: your-monitoring-token"
curl -X GET http://localhost:8787/v1/monitoring/quality \
-H "x-monitoring-token: your-monitoring-token"
curl -X GET http://localhost:8787/v1/monitoring/panel \
-H "x-monitoring-token: your-monitoring-token"
```
### 5) Admin example
```bash
curl -X POST http://localhost:8787/v1/admin/registry/servers \
-H "content-type: application/json" \
-H "x-admin-token: change-me" \
-d '{
"id":"github-main",
"name":"github-main",
"endpoint":"http://localhost:7777/mcp",
"transport":"http"
}'
```
Note: current admin registry supports only `transport=http`. `stdio` is rejected.
---
## Evaluation And Quality Gates
### Local quality baseline
```bash
npm run build
npm test
npm run smoke
```
### Token savings evaluation
```bash
npm run token-savings
npm run token-savings -- --limit 50
npm run token-savings:small
npm run token-savings:conversation:small
```
### Token savings CI gates
Linux/macOS:
```bash
TOKEN_SAVINGS_REQUIRE_BASELINE_GATES=true \
TOKEN_SAVINGS_REQUIRE_COVERAGE_GATES=true \
npm run token-savings
```
PowerShell:
```powershell
$env:TOKEN_SAVINGS_REQUIRE_BASELINE_GATES="true"
$env:TOKEN_SAVINGS_REQUIRE_COVERAGE_GATES="true"
npm run token-savings
```
### Semantic retrieval evaluation
```bash
npm run semantic-retrieval
```
Scenario-level gate:
```bash
SEMANTIC_RETRIEVAL_REQUIRE_SCENARIO_GATES=true npm run semantic-retrieval
```
---
## Operations And Runtime
### Health semantics
- `/health/liveness`: process-level liveness.
- `/health/readiness`: readiness gates (memory ratio, optional tool registry requirements, and runtime checks).
### Recommended production-like baseline
Use `configs/env.production.recommended` as your base.
Important defaults for stable operation:
- `GATEWAY_AUTH_ENABLED=true`
- `MONITORING_AUTH_ENABLED=true`
- `MOCK_EXECUTOR=false`
- `SEED_DEMO_DATA=false`
- `SESSION_MEMORY_PERSIST_ENABLED=true`
- `OPS_REQUEST_LOGGING_ENABLED=true`
### Persistence files
- Session memory: `configs/session-intent-memory.json`
- Registry state: `configs/registry-state.json`
- Router vectors: `configs/router-vectors.sqlite`
These are local single-node files and should be managed with backup/permissions in real deployment.
---
## Troubleshooting
### 1) PowerShell `curl` confusion
In PowerShell, `curl` may map to `Invoke-WebRequest`.
Use `curl.exe` or `Invoke-RestMethod` explicitly.
### 2) Auth 403 on gateway or monitoring
Check all of the following:
1. `GATEWAY_AUTH_ENABLED` and/or `MONITORING_AUTH_ENABLED` are enabled as expected.
2. Header name is correct (`x-gateway-token`, `x-monitoring-token`, `x-admin-token`).
3. Token value has no accidental spaces or quote artifacts.
4. `.env` was loaded by current process (restart service after env changes).
### 3) `.env` value parsing issues
Use this pattern in `.env`:
```env
GATEWAY_TOKEN=plain_no_space_value
```
Avoid inline comments on same line as secret values.
### 4) ByteString error in OpenAI-compatible client
If you see `Cannot convert argument to a ByteString` from `undici`, a header/env value likely contains unsupported characters.
Check token env vars and header construction for non-ASCII or malformed characters.
### 5) Intent provider mismatch
If using non-OpenAI model, ensure provider triplet is consistent:
```env
INTENT_PROVIDER=compatible
INTENT_MODEL=your-model
INTENT_API_KEY=your-key
INTENT_BASE_URL=https://your-endpoint/v1
```
### 6) Readiness failing
Check:
- memory pressure (`OPS_MAX_HEAP_USED_RATIO`)
- registry requirements (`OPS_READINESS_REQUIRE_TOOLS`)
- runtime logs for last startup warnings
---
## Security Checklist Before GitHub Publish
1. Never commit real `.env` secrets.
2. Keep `.env.example` as placeholder-only template.
3. Rotate any key that was ever printed in terminal or logs.
4. Ensure auth is enabled in shared environments:
- `GATEWAY_AUTH_ENABLED=true`
- `MONITORING_AUTH_ENABLED=true`
5. Verify generic 500 error response is sanitized.
6. Verify request body limit is configured (`HTTP_JSON_BODY_LIMIT`).
7. Keep `MOCK_EXECUTOR=false` for non-local usage.
8. Run full local checks before push:
- `npm run build`
- `npm test`
- `npm run smoke`
9. Include repository governance files:
- `LICENSE`
- `CONTRIBUTING.md`
- issue/pr templates (optional but recommended)
---
## Known Limitations
1. OpenAI embedding provider is optional; hash embedding fallback is expected when provider/key/network unavailable.
2. Policy and tenancy controls are baseline level and should be hardened for strict multi-tenant production.
3. Registry and memory persistence are local-file oriented and not built for distributed concurrent writers.
4. `stdio` MCP transport is intentionally disabled in current release.
5. SQLite vector storage currently uses JSON arrays and can be upgraded to optimized vector extension later.
---
## Notes For Community Users
This project is practical for local development, private deployments, and open-source experimentation.
If you only publish to GitHub for community collaboration (not public internet exposure), you can still keep auth enabled by default to prevent accidental misuse in shared environments.
