Content
# 📡 TelecomOS — private network intelligence
**Zero-egress AI for the NOC: root-cause analysis, alarm-storm correlation, and
human-gated self-healing for 5G networks — telemetry never leaves your machine.**
Built on [LLM OS](https://github.com/Indianinnovation/llm-os) — the model, the reasoning, and the data all stay local.
*Live, unedited: an injected backhaul storm is correlated to its **root cause**, the agent **proposes** a fix, an unauthorized user is **DENIED**, the NOC manager **approves**, execution **heals the network** (issues 6→5, alarms 9→7) — and the AI console answers through the audited kernel. Full video: [docs/dashboard_demo.mp4](docs/dashboard_demo.mp4).*
Network telemetry — KPIs, alarms, cell topology, subscriber-adjacent logs — is
regulated, sovereign data. Data-localization rules and operator security policy
make cloud LLMs a non-starter in the NOC. TelecomOS runs the entire diagnostic
and remediation loop on a **local model**, works **air-gapped**, and writes every
decision into a **tamper-evident audit chain**.
## 🎯 What it does
**Diagnose**
- **LangGraph 5-node RCA pipeline** — Classify → Retrieve → Analyze → Recommend → Validate, on a local 7B model (no API keys, flat cost)
- **Instant network-wide triage** — deterministic rules, no LLM call (10 cells in ~12 ms)
- **Alarm-storm correlation** — separates the **probable root alarm** from downstream symptoms by site topology and causal ranking
- **Cluster impact scoping** — isolated cell vs site-wide vs area-wide (dispatch the field team, or don't)
- **Trend analysis** — which KPIs are worsening, and *when* each first breached threshold
**Act (safely)**
- **Closed-loop remediation with human approval gates** — the agent may **propose**, never execute. Execution is mechanically refused unless the action is `APPROVED`, and approval requires the `noc_manager` role held by an **authenticated console operator** — the agent has no approval tool, so it cannot self-approve. Proposed → approved(by) → executed → verified is the audit record. If verification finds the cell has NOT recovered, the next catalogued action is auto-proposed — again behind the human gate — until the catalog is exhausted and it escalates. → [`docs/ENTERPRISE.md`](docs/ENTERPRISE.md)
- **Execution guardrails** — approvals expire (TTL), per-site blast-radius caps, and `maintenance.json` change-freeze windows: every check a change-management process demands, mechanical and fail-closed
- **Live network simulator** — injectable fault scenarios behind the same connector interface real OSS plugs into, so the whole loop is demonstrable end-to-end today
**Know**
- **📄 3GPP spec answers with clause citations** — offline RAG over your spec library: *"what does the spec say about RLF?"* → `3GPP TS 38.331 § 5.3.10`, air-gapped
- **📝 Incident reports** — full RCA formatted as a ticket-ready markdown document
- **⚙️ Config audit** — 7 KPI baseline checks + alarm-to-config mapping, 0–100 score
- **🧠 Incident memory** — past incident reports are indexed locally; diagnosis can cite precedent: *"same signature as INC-0042 — the fix that worked was X"*
- **🔁 Proactive monitoring** — sweeps that record every degradation/recovery; *"has this happened before?"*
**Integrate**
- **📥 Bring-your-own-data** — map any vendor KPI export (Ericsson/Nokia/Huawei column names, id formats, units) with one YAML: [`examples/vendor_mapping.example.yaml`](examples/vendor_mapping.example.yaml)
- **📡 Real-data ingestion — file drop** — point `TELECOMOS_PM_DROP_DIR` at wherever your EMS already writes periodic performance files (3GPP TS 32.435 PM-XML or CSV) and every tool reads it live, no code changes, no network stack: [`examples/pm_xml_sample.xml`](examples/pm_xml_sample.xml) · `./scripts/filedrop_demo.sh`
- **22 MCP tools** — plug into LLM OS (or any MCP host); every call digest-pinned, egress-monitored, hash-chain audited
- **Cloud strictly opt-in** — OpenAI/Bedrock still exist behind `LLM_BACKEND`, but the default cannot leak
## 🏗️ Architecture
```
NOC dashboard (Streamlit) LLM OS kernel ── chat, routing,
grid · storms · approvals (local model) audit chain,
│ │ egress sentinel,
│ │ MCP model pinning
▼ ▼
┌──────────────────────────────────────────────────────────────────┐
│ TelecomOS │
│ │
│ mcp_server.py ── 22 tools ──┬─ diagnostics (5-node RCA) │
│ ├─ triage · storms · trends │
│ ├─ spec RAG (cited) │
│ └─ remediation: propose → APPROVE │
│ (RBAC) → execute │
│ │ │
│ connectors/ ◄────────────────────┘ │
│ • SimulatorConnector (shipped) │
│ • FileDropConnector — PM-XML/CSV + alarm-file ingestion (shipped)│
│ • SNMP · Kafka · OSS execution (dry-run-first) (shipped) │
│ │ │
│ data: KPI CSV · alarms JSON · logs · topology · specs │
└──────────────────────────────────────────────────────────────────┘
everything on this machine
```
## 🖥️ The NOC dashboard
```bash
./scripts/run_dashboard.sh storm # or: congestion | (no arg = recorded data)
```
A dark console at **localhost:8502** (loopback-only — nothing is exposed on your network):
| View | What it shows |
|---|---|
| **Network** | Site/cell grid with live status + PRB / throughput / BLER trend charts |
| **Alarms & storms** | Active alarms, and storms correlated to a probable root cause |
| **Remediation queue** | The closed loop: propose → **approve (RBAC-gated)** → execute → **verify**, with the network healing in the stat tiles |
| **AI console** | Chat through the LLM OS kernel; every answer shows the tool used and its audit id |
Switch identity in the sidebar (`viewer1` has no roles — watch approval get
**denied**; `dilip` is a `noc_manager`) and drive live faults with the simulation
controls. The sidebar also surfaces the kernel's **egress sentinel** and
**model-pinning** status.
## 🔌 Run it inside LLM OS (recommended)
Add to your LLM OS `mcp_servers.json` (absolute paths):
```json
"telecom-diagnostics": {
"command": "/path/to/telecomos/.venv/bin/python",
"args": ["/path/to/telecomos/mcp_server.py"]
}
```
The kernel discovers all 22 tools and routes NOC questions to them:
| Ask the chat… | Routed to | Speed |
|---|---|---|
| "Which cells have problems?" | `network_health_check` (deterministic) | ~12 ms |
| "Why is cell 3 degraded?" | `diagnose_cell` (full 5-node RCA) | ~40 s |
| "Correlate these alarms — what's the root cause?" | `correlate_alarm_storm` | instant |
| "Propose a fix for cell 2" | `propose_remediation` (needs approval) | instant |
| "Execute ACT-123" (after a human approves in the console) | `execute_remediation` | instant |
| "Did the fix work?" | `verify_remediation` (proposes a gated follow-up if not) | instant |
| "Approve ACT-123" | *(no agent tool — approval is a human action in the NOC console)* | — |
| "What does the spec say about RLF?" | `search_specs` (cited: TS + clause) | instant |
| "Is it just cell 3 or the whole site?" | `analyze_cluster_impact` | instant |
| "When did cell 3 start degrading?" | `get_kpi_trend` | instant |
| "Anything new since the last check?" | `monitor_sweep` | instant |
| "Has cell 3 degraded before?" | `get_status_history` | instant |
| "Write an incident report for cell 7" | `generate_incident_report` | ~40 s |
| "Audit the config of cell 1" | `audit_cell_config` | instant |
| "Start the backhaul_storm simulation" | `start_simulation` · `stop_simulation` | instant |
| "Is the PM file drop working?" | `file_drop_status` | instant |
| "Has this happened before?" (precedent) | `find_similar_incidents` | instant |
| "What's about to break?" (prediction) | `forecast_cell` | instant |
| KPIs / alarms / logs for a cell | `get_cell_kpis` · `get_active_alarms` · `get_recent_logs` | instant |
Cell ids are fuzzy: "cell 3", "CELL_003" and "3" all resolve.
Real answer from the pipeline, fully offline:
> **You:** Why is cell 3 degraded? ⚙ *routed to `diagnose_cell` · audited*
> **Agent:** Root cause: a traffic spike drove RRC connected users to 265 and
> PRB utilization to 91%, collapsing throughput to 8 Mbps (congestion,
> confidence 0.99). Recommended: traffic shaping + QoS prioritization, …
## ✈️ Prove it: air-gapped
*Wi-Fi radio disabled **on camera** → 10-cell triage → full RCA on cell 3 → alarm
drill-down → audit chain verified, hashes on screen. Reproduce: `./scripts/noc_demo.sh`
(kills the Wi-Fi radio itself). Video: [docs/noc_demo.mp4](docs/noc_demo.mp4).*
The closed-loop demo — fault → propose → denied → approved → executed → healed —
is scripted too: `./scripts/selfheal_demo.sh`.
## 📊 Routing accuracy (measured, not claimed)
Scored with the [LLM OS eval harness](https://github.com/Indianinnovation/llm-os)
on a 24-prompt telecom set, including trap questions that must **not** trigger
diagnostics (definitional questions may answer directly or cite the spec library
— both count as correct):
| local model | telecom routing accuracy |
|---|---|
| qwen2.5-coder (7B) | **96%** |
| llama3.2 (3B) | 88% |
## 🚀 Quickstart (standalone, fully local)
Requires [Ollama](https://ollama.com) — no API keys:
```bash
git clone https://github.com/Indianinnovation/telecomos.git
cd telecomos
ollama pull qwen2.5-coder # the local reasoning model
ollama pull all-minilm # embeddings for the spec RAG
python3 -m venv .venv && source .venv/bin/activate
pip install -r requirements.txt
# Run the engine hardened: loopback-only, vendor cloud features OFF
OLLAMA_HOST=127.0.0.1:11434 OLLAMA_NO_CLOUD=1 ollama serve # (see LLM OS README)
./scripts/run_dashboard.sh storm # NOC dashboard → localhost:8502
uvicorn app.main:app --port 8000 # optional: the REST API
```
To *deliberately* use a cloud backend: `LLM_BACKEND=openai` with `OPENAI_API_KEY`
(and `pip install langchain-openai`). The default backend is local and cannot leak.
## 🧪 Sample scenarios (10 cells)
| Scenario | Cell | Expected issue |
|---|---|---|
| Throughput drop | CELL_001 | Congestion |
| High BLER | CELL_002 | BLER issue |
| Cell congestion | CELL_003 | Congestion |
| Low SINR | CELL_005 | Interference |
| Link failure | CELL_007 | Link failure |
| Power degradation | CELL_009 | Hardware fault |
Live fault injection: `start_simulation("cell_congestion")` or `("backhaul_storm")`
— telemetry drifts, alarms fire, and approved remediations actually resolve them.
## 📁 Project structure
```
telecomos/
├── app/
│ ├── connectors/ # NetworkConnector contract + live simulator
│ ├── graph/agent.py # LangGraph 5-node RCA (local-first LLM)
│ ├── tools/ # kpi · alarms · logs · triage · trends · cluster
│ │ # · storms · monitor · reports · config · spec RAG
│ │ # · remediation (approval gates + RBAC)
│ ├── api/routes.py # FastAPI endpoints
│ └── main.py
├── data/ # sample telemetry · topology · rbac
├── specs/ # 3GPP spec library (replace with official docs)
├── docs/ENTERPRISE.md # safety model · SSO · fleet · connector roadmap
├── mcp_server.py # 22 MCP tools → LLM OS
├── dashboard.py # the NOC console
└── scripts/ # run_dashboard · noc_demo · selfheal_demo
```
## 🏢 Enterprise
[`docs/ENTERPRISE.md`](docs/ENTERPRISE.md) covers the closed-loop safety model,
SSO/OIDC identity (production roles come from the session, never from chat), fleet
deployment, and the connector contract for SNMP / Kafka / 3GPP PM-XML / OSS
execution — with an honest list of what is shipped versus what awaits a design
partner's environment.
## 🛠️ Tech stack
- **LLM OS** — local agentic kernel: routing, audit chain, model pinning, egress sentinel
- **Ollama / qwen2.5-coder** — local reasoning (cloud strictly opt-in)
- **LangGraph** — RCA pipeline orchestration
- **ChromaDB** — offline spec index (telemetry disabled by policy)
- **FastAPI · Streamlit · Plotly · Pandas**
## 🧭 Why "OS"?
TelecomOS is the telecom vertical of the [LLM OS](https://github.com/Indianinnovation/llm-os)
platform: the same auditable local kernel, loaded with telecom organs. Same
pattern, any regulated industry.
## 📜 License
MIT
