KnowledgeForge

<p align="center"> <img src="docs/assets/banner.png" alt="KnowledgeForge Banner" width="100%"/> </p> <h1 align="center">KnowledgeForge</h1> <p align="center"> <strong>Enterprise Agent Operating System (AOS)</strong> <br/> <em>Secure · Multi-tenant · Autonomous Swarms · Continuous Empirical Learning · Production-grade</em> </p> <p align="center"> <img src="https://img.shields.io/badge/python-3.11+-blue?logo=python" alt="Python 3.11+"/> <img src="https://img.shields.io/badge/FastAPI-0.115-009688?logo=fastapi" alt="FastAPI"/> <img src="https://img.shields.io/badge/LangGraph-AOS_orchestration-purple?logo=langchain" alt="LangGraph"/> <img src="https://img.shields.io/badge/LiteLLM-universal_gateway-green" alt="LiteLLM"/> <img src="https://img.shields.io/badge/mem0-cloud_memory-blueviolet" alt="mem0"/> <img src="https://img.shields.io/badge/tests-781_passing-brightgreen" alt="Tests"/> <img src="https://img.shields.io/badge/RAGAS-evaluated-orange" alt="RAGAS"/> <img src="https://img.shields.io/badge/license-MIT-blue" alt="MIT License"/> </p> --- ## About **KnowledgeForge** is a high-performance, enterprise-grade **Agent Operating System (AOS)** designed for secure, multi-tenant agentic automation and knowledge synthesis. Moving beyond static RAG architectures, KnowledgeForge orchestrates autonomous specialized swarms, manages sandboxed code execution, schedules background daemons, and implements continuous empirical learning for autonomous skill evolution. **Key Architecture Pillars:** - 🏢 **Multi-Tenant Isolation & RBAC** — Cryptographically isolated tenant vaults, Department-level workspace segmentation, and granular Role-Based Access Control. - 🤖 **Autonomous Swarm Orchestration** — Built on LangGraph to coordinate specialized agents (ReAct Loops, Deep Lookup, Subtask Swarms, RAG Specialists, and Sandbox Execution Runners). - 🧠 **Continuous Empirical Learning (CEL) Engine** — Inspired by human cognitive psychology, where the deepest, most permanent behavioral adjustments occur in response to mistakes, fear, or critical failures. Rather than just learning from positive reinforcements, the CEL Engine actively monitors execution "traumas" (exceptions, tool failures, and logical bottlenecks). It translates these failures into Direct Preference Optimization (DPO) preference rules, dynamically refining the agent's steering policy. Candidate code skills are synthesized and consolidated using a sandboxed AST merger, enabling the system to evolve organically from its own errors. Detailed guide in [Empirical Learning Documentation](docs/self-learning/README.md). - 🔒 **Hardened Sandbox Execution** — Rootless Docker/Firecracker microVM executions with real-time secret scrubbers and silence watchdogs to safely run untrusted code. - 📡 **Reactive SSE Gateway & Daemons** — Concurrent background worker daemons (scheduler, state archiver) integrated with a Server-Sent Events (SSE) PubSub channel for real-time state sync. - 🔌 **Universal Inference Gateway** — Zero-code upstream switching to any provider (Ollama, vLLM, Gemini, OpenRouter, Anthropic, Claude, DeepSeek) using token-aware KV caching. - 📈 **Full Observability & Telemetry** — OpenTelemetry tracing, Prometheus metrics, and automated RAGAS A/B Prompt evaluation pipelines. --- ## System Architecture <p align="center"> <img src="docs/images/architecture.png" alt="KnowledgeForge System Architecture" width="100%"/> </p> KnowledgeForge is designed as an enterprise-grade AI knowledge platform built on a robust, scalable, and secure architecture. The system data flow and request processing pipeline is shown in the architecture diagram above. ### 1. The 5-Layer Context Engineering Engine <p align="center"> <img src="docs/images/context_memory_flow.png" alt="5-Layer Context Engineering and Multi-Tier Memory Flow" width="100%"/> </p> Managing long-context windows efficiently while preserving focus is handled by the context builder inside [src/memory/working.py](src/memory/working.py). It structures context into 5 distinct, budget-controlled layers to mitigate "Lost in the Middle" attention degradation: - **Layer 1: System Instructions & Safety Boundaries (L1)**: Static, cached system instructions, Markdown formatting schemas, and tool-calling constraints. Placed at the very top. - **Layer 2: Personalization & User Facts Profile (L2)**: Dynamically recalled user facts and preferences retrieved from `mem0` memory. Prefixed with `[User profile]` delimiters. - **Layer 3: Document/RAG Retrieval Context (L3)**: Relevance-ranked passages from the hybrid retrieval pipeline. Chunks are reordered via `_lost_in_middle_reorder` (most relevant chunks placed at the start and end of the block, less relevant chunks in the middle) to maximize attention. - **Layer 4: Current User Query & Instruction (L4)**: Placed at the absolute bottom of the context, closest to the generation target, which ensures the model prioritizes the active task instructions. - **Layer 5: Conversation & Session History (L5)**: Sliding-window memory. Older history turns are automatically compressed into key bullets via LLM summarization (`summarize_history`), while the most recent turn-pairs are preserved verbatim to maintain local dialog coherence. ### 2. Multi-Tier Agent Memory System KnowledgeForge features a tiered, multi-tier agent memory registry to emulate human cognitive recall: | Memory Tier | Scope | Backend | Lifecycle | Key Capabilities | |-------------|-------|---------|-----------|------------------| | **Working Memory** | Session-level | In-Memory / Redis | Active thread session | Token counting, sliding window pruning, token usage warnings, history summarization. | | **Semantic Memory** | Cross-session | mem0 Cloud (Vector Index) | Persistent (survives restarts) | Fact extraction, user profile accumulation, contradiction resolution (e.g. updating old facts), relevance score thresholding ($\ge 0.15$). | | **Episodic Memory**| Historical overview | MongoDB / File Store | Background async consolidation | Parses conversation chains asynchronously to extract high-level episodes and store session summaries. | --- ## Features ### Advanced RAG Pipeline <p align="center"> <img src="docs/images/retrieval_ingestion_pipeline.png" alt="Advanced Retrieval and Ingestion Pipeline" width="100%"/> </p> KnowledgeForge contains a highly optimized, enterprise-grade RAG pipeline that implements state-of-the-art retrieval and generation techniques: - **Lexical & Semantic Ensemble (BM25 + Vector Search)**: Combines dense vector semantic search (Qdrant/Chroma) and lexical keyword matching (Rank-BM25), fusing candidate rankings with Reciprocal Rank Fusion (RRF, constant $k=60$) to balance exact keyword hits with conceptual understanding. - **Cross-Encoder Reranking (`ms-marco-MiniLM-L-6-v2`)**: Acts as a secondary relevance filter by scoring document chunks directly against the user query. This process discards low-quality retrieved passages, significantly compressing the prompt context size and reducing downstream token costs. - **Corrective RAG (CRAG)**: Employs a dedicated LLM relevance grader to review the retrieved document chunks. If chunk confidence falls below the acceptance threshold, the grader dynamically triggers external web search tool fallbacks to resolve knowledge gaps and eliminate hallucinations. - **Graph RAG Integration (LightRAG)**: Combines standard vector RAG with LightRAG. It indexes and queries entities and relationship graphs scoped by organization and department boundaries, enabling multi-hop reasoning over complex themes and structural connections that traditional vector chunking fails to locate. - **Hypothetical Document Embeddings (HyDE)**: Pre-generates a hypothetical answer paragraph matching the query, then uses its embedding to perform semantic searches, bridging the semantic gap between questions and raw document statements. - **Scope-Isolated Semantic Caching**: Caches previous responses based on query cosine similarity $\ge 0.92$. To prevent data leakage, the cache is isolated by user role and department boundaries. - **Context-Aware Query Rewriting**: Uses conversation history and the latest user prompt to rewrite query inputs into clear standalone questions, resolving coreferences, pronouns, and implicit context. - **Multi-Tenant & RBAC Filtered Retrieval**: Ingests and queries documents with organization, department, and role metadata filters applied at the database level, guaranteeing absolute data isolation between departments. - **SSE streaming**: Real-time token delivery via `POST /api/v1/chat/stream`. ### Multi-Agent (LangGraph) KnowledgeForge implements a state-of-the-art **centralized stateful orchestrator** using LangGraph, shifting the architecture from isolated specialist pipelines to a unified reasoning kernel. <p align="center"> <img src="docs/images/orchestration_routing.png" alt="LangGraph Multi-Agent Orchestration & Intent Routing Loops" width="100%"/> </p> Set `RAG_MODE=agent` to activate the full agentic loop: - **Intent Router** — `semantic-router` embedding classification (categorizes into direct, rag, tool, or multi-agent lanes). - **Coordinator Node** — Central dispatcher that initializes trajectory variables and manages dynamic model cascading. - **Planner Node** — Decomposes complex natural language goals into a structured DAG (Directed Acyclic Graph) task queue. - **Reasoning Loop Nodes**: - `react_reason` — Formulates the next action step, using historical trajectory contexts and error tracebacks. - `react_act` — Compiles and validates execution payload variables. - `react_observe` — Receives, parses, and encapsulates execution outcomes. - **RAG & Deep Research Nodes** — Executes lexical-semantic hybrid queries, cross-encoder rerankings, and multi-hop recursive retrieval grading. - **Tool Executor Node** — Interfaces with sandboxed environments, dynamic tools, and multiplexed remote MCP server channels. - **QA Validator Node** — Runs post-generation hallucination checks, verifying synthesized answers against retrieved source documents. ### Auth & Multi-Tenancy - JWT authentication with refresh tokens - Organization → Department → User hierarchy - Role-based access: `admin` | `member` | `viewer` - API versioning: `/api/v1/*` with backward compat - Rate limiting (slowapi, configurable) ### Observability <p align="center"> <img src="docs/images/observability_infrastructure.png" alt="Infrastructure, Observability and Telemetry Stack" width="100%"/> </p> | Provider | Purpose | Toggle | |----------|---------|--------| | **LangSmith** | Full pipeline trace spans with I/O | `OBSERVABILITY_PROVIDER=langsmith` | | **Langfuse** | Self-hosted tracing + analytics | `OBSERVABILITY_PROVIDER=langfuse` | | **Prometheus** | 11 metric types (latency, tokens, cost, cache) | Always active at `/metrics` | ### Document Understanding - **10 formats**: PDF, DOCX, PPTX, XLSX, TXT, MD, HTML, JPG, PNG, WEBP - **Docling** (IBM) — Layout-aware parsing: tables, formulas, multi-column - **Vision LLM** — Images without text → Gemini/OpenRouter describes content - **Async processing** — Non-blocking with job tracking - **SHA-256 dedup** — Skip re-ingestion of identical files --- ## Advanced Capabilities & Architecture Deep-Dive <p align="center"> <img src="docs/images/enterprise_governance_security.png" alt="Multi-Tenant RBAC & Enterprise Governance" width="100%"/> </p> KnowledgeForge is built from the ground up for production stability, cost control, strict tenant security, and high retrieval accuracy. ### 1. Resiliency & Failure Recovery - **Async Circuit Breakers**: Protects external LLM APIs from cascading failures using a custom async circuit breaker (`AsyncCircuitBreaker` in [src/core/errors.py](src/core/errors.py#L30)). If failures cross a threshold (e.g. 5 failed calls), the breaker trips to `OPEN`, immediately rejecting outgoing requests for a recovery timeout (30 seconds) to prevent server thread starvation. - **Exponential Backoff with Full Jitter**: Transient network errors or rate limits (e.g., HTTP `429 Too Many Requests`) are handled with exponential backoff capped at a maximum delay. It injects a randomized delay variable (Full Jitter algorithm) to eliminate the "thundering herd" concurrency problem on downstream servers. ### 2. Quota Management & Cost Control - **Complexity-Based Query Routing**: Analyzes query structure, sentence clauses, and question triggers via the `ModelComplexityRouter` in [src/governance/cost.py](src/governance/cost.py). Simple chitchat or direct instructions are automatically routed to low-cost local models (e.g., Ollama / local Qwen), while complex multi-step analytical queries are sent to premium cloud models. - **USD & Token Budget Manager**: The `TokenBudgetTracker` monitors daily usage. It tracks accumulated token consumption and dollar spend on a per-user basis. If a request would breach the daily threshold, it raises a `TokenBudgetExceededError` before calling downstream APIs, protecting host budgets. - **Custom Model Cost Handling**: Wrapped all budgeting callbacks in defensive `try-except` blocks. If custom unmapped model gateways (like 9Router Cloudflare free pools) return completions without defined token costs, it logs a warning instead of throwing fatal errors. ### 3. Agentic Orchestration & Tool Harness - **LangGraph State Machine**: Models the agent pipeline as a directed graph. Features intent classification, planning nodes, tool executor loops, synthesis nodes, and QA reflection nodes. Includes a loop-guard counter to prevent infinite reflection loops. - **Automatic Function-to-Tool Compiler**: The `ToolRegistry` compiles standard Python functions into JSON tool schemas for LLM function calling by extracting docstrings, parameter type hints, and validation constraints at runtime. - **Model Context Protocol (MCP)**: Fully supports MCP. Interacts with external filesystem, database, or API tools over stdio and SSE transport protocols. ### 4. Enterprise Security, Privacy & Isolation - **Presidio PII Masking**: Integrates Microsoft Presidio at the API gateway. Automatically detects and redacts emails, phone numbers, credit card numbers, SSNs, and IP addresses before sending context to public LLM APIs. - **Input & Output Guardrails**: Blocks malicious prompt injection attempts (e.g., "ignore previous instructions") at the API boundary, returning HTTP `403 Forbidden`. Outputs are scanned against Vietnamese/English harmful content lists. - **KMS Secret Wrapping**: Protects sensitive adapter credentials (API keys, database connections) using secure KMS envelope decryption wrappers. - **RBAC Department Isolation**: Department metadata filters are injected into all vector search and document ingestion queries, guaranteeing that users can never retrieve documents outside their authorized department boundary. --- ## Project Structure ``` knowledgeforge/ ├── src/ # Application source code │ ├── api/ # Delivery & API Presentation layer │ │ ├── main.py # FastAPI application factory + lifespan │ │ ├── middleware.py # Rate limiters, CORS, security headers │ │ ├── deps.py # Composition Root (Dependency Injection wiring) │ │ ├── gateway/ # SSE real-time pubsub events gateway │ │ └── routes/ # FastAPI endpoint routers (auth, chat, ingest, daemons...) │ ├── core/ # Shared Domain Core (entities & abstractions) │ │ ├── interfaces/ # DIP Core Boundary Interfaces (LLM, Storage, Memory...) │ │ ├── daemons/ # Background worker daemons (CronScheduler, StateArchiver) │ │ ├── execution/ # Active execution pools & run cancellation runners │ │ ├── security/ # Tool gateway containment, regex & sandbox rules │ │ ├── dto/ # Decoupled route payload validation DTOs │ │ ├── models/ # Domain entities (auth, chat, documents...) │ │ ├── ports/ # Legacy interface shims (backward compatible) │ │ ├── context_composer.py # Context window manager │ │ └── sandbox.py # Local subprocess limits & docker sandboxes │ ├── adapters/ # Concrete Infrastructure Adapters │ │ ├── llm/ # LiteLLM universal adapter & cost trackers │ │ ├── embedding/ # Local (SentenceTransformer) & TEI clients │ │ ├── reranker/ # Cross-encoder re-ranking client │ │ ├── vectordb/ # ChromaDB (dev) and Qdrant (production) clients │ │ ├── cache/ # Redis semantic cache adapter │ │ ├── pubsub/ # Redis & In-Memory pubsub real-time event brokers │ │ ├── queue/ # Kafka-based event streams & DB queue adapters │ │ └── stores/ # Postgres, SurrealDB, SQLite & MongoDB stores │ ├── agents/ # Domain: Agentic Planning & Specialists │ │ ├── orchestrator/ # LangGraph state machine graph execution & routing lanes │ │ └── specialists/ # Choreographed domain specialist services (lightrag, landmark) │ ├── rag/ # Domain: Retrieval-Augmented Generation algorithms │ │ ├── ingestion/ # Loader, chunker, and document hash managers │ │ ├── pipeline/ # RAG steps (HyDE, CRAG grading, rewriters) │ │ └── retrievers/ # BM25 + vector hybrid fusion retrievers │ ├── memory/ # Domain: Multi-tier Personalization Memory │ │ ├── working.py # Short-term thread memory assembly │ │ ├── semantic.py # mem0 personalization and profile recall │ │ └── episodic.py # Background episodic summarization tasks │ └── tools/ # Domain: Tool Integrations & MCP │ ├── gateway.py # Bilingual security safety gates & budget controls │ ├── registry.py # Unified function-to-tool compilers │ ├── builtin.py # Internal tools (calculator, calendar) │ └── mcp/ # Model Context Protocol session pools & servers ├── config/ # External configuration files │ └── routes.yaml # Intent router utterances + thresholds ├── tests/ # Comprehensive Test Suite │ ├── unit/ # Unit tests (mocked dependencies) │ ├── integration/ # Ephemeral database integration tests │ └── ... # + api, auth, vectordb, tools, ingest ├── eval/ # Evaluation & benchmarks (restructured) │ ├── dataset.json # Evaluation dataset file │ ├── run_ragas.py # Ragas evaluation scorer │ ├── test_full_e2e.sh # E2E execution script │ └── suites/ # CI eval, Locust load tests, extra suites ├── infra/ # Infrastructure & deployment config │ └── docker/ # Dockerfile & dev/prod Docker Compose files ├── observability/ # Production monitoring configurations │ ├── prometheus/ # Prometheus rules & configs │ ├── logs/ # Loki & Promtail configuration ymls │ └── dashboards/ # Grafana analytics dashboard jsons ├── .github/workflows/ci.yml # GitHub Actions CI/CD workflows └── uv.lock # Lockfile (reproducible builds) ``` --- ## Quick Start ### Prerequisites - Python 3.11+ - [uv](https://docs.astral.sh/uv/) (package manager) - One of: Ollama / Gemini API key / OpenRouter API key ### Installation ```bash # 1. Clone & install git clone https://github.com/vtnguyen04/KnowledgeForge.git && cd KnowledgeForge uv sync # 2. Configure cp .env.example .env # Edit .env — set your LLM provider (see table below) # 3. Start local database & cache infrastructure (Docker) docker compose -f infra/docker/docker-compose.dev.yml up -d # 4. Seed demo data uv run python -m src.scripts.seed_demo # 5. Start background worker daemons (in separate terminals or screen sessions) uv run python -m src.core.daemons.run --daemon scheduler uv run python -m src.core.daemons.run --daemon archiver # 6. Start the FastAPI API server uv run uvicorn src.api.main:app --reload --port 8000 ``` ### Try it out ```bash # Register & login curl -X POST http://localhost:8000/api/v1/auth/register \ -H "Content-Type: application/json" \ -d '{"email":"demo@test.com","password":"pass123","name":"Demo User","org_name":"TechViet"}' TOKEN=$(curl -s -X POST http://localhost:8000/api/v1/auth/login \ -H "Content-Type: application/json" \ -d '{"email":"demo@test.com","password":"pass123"}' | jq -r '.access_token') # Ask a question curl -X POST http://localhost:8000/api/v1/chat \ -H "Authorization: Bearer $TOKEN" \ -H "Content-Type: application/json" \ -d '{"query": "What is the PTO policy for engineers?"}' ``` --- ## LLM Providers Switch provider by changing `LLM_MODEL` in `.env`. **Zero code changes.** | Provider | `LLM_MODEL` | `LLM_API_BASE` | Cost | |----------|-------------|----------------|------| | **Ollama** (local) | `ollama/qwen3:1.7b` | `http://localhost:11434` | Free | | **Gemini** (cloud) | `gemini/gemini-2.5-flash` | _(empty)_ | Free tier | | **OpenRouter** (cloud) | `openrouter/qwen/qwen3-1.7b:free` | _(empty)_ | Free | | **vLLM** (GPU) | `hosted_vllm/Qwen/Qwen3-1.7B` | `http://localhost:8100/v1` | Self-hosted | | **Anthropic** | `anthropic/claude-sonnet-4-20250514` | _(empty)_ | Pay-per-use | | **OpenAI** (cloud) | `openai/gpt-4o-mini` | _(empty)_ | Pay-per-use | | **9Router** (gateway) | _(empty)_ | `http://localhost:20128/v1` | Free (with fallback) | **Automatic fallback:** Set `LLM_FALLBACK_MODELS=model1,model2` — switches on rate limit errors. --- ## API Reference ### Authentication (`/api/v1/auth/*`) | Method | Endpoint | Description | |--------|----------|-------------| | `POST` | `/api/v1/auth/register` | Register a new user and organization. Payload: email, password, name, org_name. | | `POST` | `/api/v1/auth/login` | Authenticate user and return JWT access and refresh tokens. Payload: email, password. | | `POST` | `/api/v1/auth/refresh` | Obtain a new access token using a valid refresh token. Payload: refresh_token. | | `GET` | `/api/v1/auth/me` | Retrieve the authenticated user's profile details. | | `PATCH`| `/api/v1/auth/profile` | Update user profile fields (e.g. name, preferences). | | `POST` | `/api/v1/auth/logout` | Invalidate current session and refresh tokens. | ### Chat & Agentic Queries (`/api/v1/chat/*`) | Method | Endpoint | Description | |--------|----------|-------------| | `POST` | `/api/v1/chat` | Send a query to the standard synchronous RAG/Agent pipeline. Payload: query, session_id, config. | | `POST` | `/api/v1/chat/stream` | Send a query to the streaming SSE (Server-Sent Events) pipeline. Payload: query, session_id, config. | ### Document Ingestion (`/api/v1/ingest/*`) | Method | Endpoint | Description | |--------|----------|-------------| | `POST` | `/api/v1/ingest/upload` | Upload a document file (multipart form) for RAG parsing and chunk indexing. | | `POST` | `/api/v1/ingest/ingest` | Trigger background ingestion for a local file path or directory. Payload: path. | | `POST` | `/api/v1/ingest/web` | Ingest a remote web page URL into the RAG database. Payload: url. | | `GET` | `/api/v1/ingest/jobs` | Retrieve a list of recent background ingestion jobs and their statistics. | | `GET` | `/api/v1/ingest/jobs/{job_id}` | Query the processing status (pending, processing, completed, failed) of a specific ingestion job. | ### Conversation Sessions (`/api/v1/sessions/*`) | Method | Endpoint | Description | |--------|----------|-------------| | `GET` | `/api/v1/sessions` | List all historical chat sessions for the current user (paginated). | | `GET` | `/api/v1/sessions/{session_id}` | Retrieve full message thread history for a specific conversation session. | | `DELETE`| `/api/v1/sessions/{session_id}` | Permanently delete a conversation session and its associated message history. | ### Landmark Catalog CRUD (`/api/v1/landmarks/*`) | Method | Endpoint | Description | |--------|----------|-------------| | `POST` | `/api/v1/landmarks` | Create a new landmark metadata entry (Admin/Member only). | | `GET` | `/api/v1/landmarks` | Retrieve and search a list of all landmark catalog entries. | | `GET` | `/api/v1/landmarks/{landmark_id}` | Get detailed metadata for a specific landmark catalog entry. | | `PUT` | `/api/v1/landmarks/{landmark_id}` | Update metadata fields of an existing landmark entry. | | `DELETE`| `/api/v1/landmarks/{landmark_id}` | Delete a landmark entry from the database catalog. | ### Feedback & Safety (`/api/v1/feedback/*`) | Method | Endpoint | Description | |--------|----------|-------------| | `POST` | `/api/v1/feedback` | Submit thumbs up/down user feedback for a specific chat response. Payload: message_id, score, comment. | ### Administration (`/api/v1/admin/*` — Requires `role=admin`) | Method | Endpoint | Description | |--------|----------|-------------| | `GET` | `/api/v1/admin/analytics` | Retrieve system-wide token costs, daily USD budgeting stats, and top chat topics. | | `GET` | `/api/v1/admin/users` | List all registered users across organizations (paginated). | | `GET` | `/api/v1/admin/documents` | List all ingested documents and their tenant metadata scopes. | | `GET` | `/api/v1/admin/audit-log` | Access read-only security audit trails logging system and user activities. | | `GET` | `/api/v1/admin/health` | Query detailed status of all databases, LLM endpoints, and local GPU containers. | | `DELETE`| `/api/v1/admin/cache` | Purge all semantic Redis cache and LiteLLM response caches. | | `GET` | `/api/v1/admin/prompts` | List all versioned prompt templates registered in the central prompt hub. | | `GET` | `/api/v1/admin/prompts/{name}` | Retrieve template details for a named prompt. | | `PUT` | `/api/v1/admin/prompts/{name}` | Add or update a prompt template override. | | `DELETE`| `/api/v1/admin/prompts/{name}` | Revert custom overrides for a prompt template to its default instructions. | | `GET` | `/api/v1/admin/few-shot` | List all registered few-shot QA examples. | | `POST` | `/api/v1/admin/few-shot` | Register a new few-shot QA example in the semantic vector collection. | | `DELETE`| `/api/v1/admin/few-shot/{example_id}` | Remove a few-shot QA example by ID. | ### System Checks & Metrics | Method | Endpoint | Description | |--------|----------|-------------| | `GET` | `/health` | Basic application liveness check (used by Docker liveness checks). | | `GET` | `/metrics` | Export Prometheus telemetry metrics (latencies, token counts, error rates, costs). | --- ## Evaluation ### RAGAS Benchmark (Qwen3-1.7B, 20 samples) | Metric | Score | Description | |--------|-------|-------------| | **Context Precision** | **1.0000** 🏆 | Retrieved chunks are highly relevant | | **Answer Relevancy** | **0.7820** | Answers address the question | | **Faithfulness** | **0.7500** | Answers grounded in retrieved context | | **Context Recall** | **0.7625** | Important context is retrieved | ```bash # Run full RAGAS evaluation uv run python eval/run_ragas.py # With custom judge model uv run python eval/run_ragas.py --judge gemini/gemini-2.5-flash # Load testing uv run locust -f eval/locustfile.py --host=http://localhost:8000 ``` ### Performance Benchmarks | Provider | Avg Latency | Recall | Retrieval Accuracy | |----------|-------------|--------|-------------------| | vLLM (Qwen3-1.7B, RTX 3060) | **595ms** | 53.6% | 95% | | Ollama (qwen2.5:1.5b) | 2085ms | 50.5% | 95% | | OpenRouter (Ling 1T) | 2982ms | 51.3% | 100% | --- ## Docker ```bash # Development (lightweight — app + Ollama) docker compose -f docker-compose.dev.yml up # Production (full stack — 12 services) docker compose -f docker-compose.prod.yml up ``` **Production services:** API · vLLM · TEI · Qdrant · MongoDB · Redis · Prometheus · Grafana · Loki · Promtail · Langfuse · Langfuse-DB --- ## Testing ```bash # Unit tests (781 tests, mocked dependencies) uv run pytest tests/ -v # E2E tests (29 tests, requires running server) bash eval/test_full_e2e.sh # RAGAS evaluation (20 questions, 4 metrics) uv run python eval/run_ragas.py ``` | Suite | Count | Status | |-------|-------|--------| | Unit tests | 781 | ✅ Passing | | E2E tests | 29 | ✅ Passing | | RAGAS eval | 20 questions × 4 metrics | ✅ Baselined | --- ## Configuration All settings via `.env` file. See [`.env.example`](.env.example) for full reference. | Variable | Default | Description | |----------|---------|-------------| | `LLM_MODEL` | `ollama/llama3.2:1b` | LLM provider + model | | `LLM_API_BASE` | _(empty)_ | API base for self-hosted LLMs | | `LLM_FALLBACK_MODELS` | _(empty)_ | Comma-separated fallback chain | | `VECTORDB_BACKEND` | `chroma` | `chroma` or `qdrant` | | `STORE_BACKEND` | `memory` | `memory` or `mongodb` | | `OBSERVABILITY_PROVIDER` | `none` | `langsmith`, `langfuse`, or `none` | | `RAG_MODE` | `single` | `single` or `agent` (LangGraph) | | `ENABLE_HYDE` | `false` | HyDE query expansion | | `ENABLE_CRAG` | `false` | Corrective RAG grading | | `RATE_LIMIT` | `60/minute` | API rate limit | --- ## Tech Stack <table> <tr> <td><b>Category</b></td> <td><b>Technologies</b></td> </tr> <tr> <td><b>🧠 LLM Orchestration</b></td> <td> <img src="https://img.shields.io/badge/LiteLLM-black?logo=data:image/svg+xml;base64,PHN2ZyB4bWxucz0iaHR0cDovL3d3dy53My5vcmcvMjAwMC9zdmciIHZpZXdCb3g9IjAgMCAyNCAyNCI+PHBhdGggZD0iTTEyIDJMMiAyMmgyMEwxMiAyeiIgZmlsbD0iI2ZmZiIvPjwvc3ZnPg==&logoColor=white" alt="LiteLLM"/> <img src="https://img.shields.io/badge/LangGraph-purple?logo=langchain&logoColor=white" alt="LangGraph"/> <img src="https://img.shields.io/badge/LangChain-green?logo=langchain&logoColor=white" alt="LangChain"/> <img src="https://img.shields.io/badge/semantic--router-blue" alt="semantic-router"/> </td> </tr> <tr> <td><b>🔌 LLM Providers</b></td> <td> <img src="https://img.shields.io/badge/vLLM-GPU-red?logo=nvidia&logoColor=white" alt="vLLM"/> <img src="https://img.shields.io/badge/Ollama-local-black?logo=ollama&logoColor=white" alt="Ollama"/> <img src="https://img.shields.io/badge/Gemini-blue?logo=google&logoColor=white" alt="Gemini"/> <img src="https://img.shields.io/badge/OpenRouter-orange" alt="OpenRouter"/> <img src="https://img.shields.io/badge/Anthropic-brown?logo=anthropic&logoColor=white" alt="Anthropic"/> <img src="https://img.shields.io/badge/9Router-gateway-teal" alt="9Router"/> </td> </tr> <tr> <td><b>🗄️ Vector & Storage</b></td> <td> <img src="https://img.shields.io/badge/Qdrant-DC382D?logo=qdrant&logoColor=white" alt="Qdrant"/> <img src="https://img.shields.io/badge/ChromaDB-orange" alt="ChromaDB"/> <img src="https://img.shields.io/badge/MongoDB-47A248?logo=mongodb&logoColor=white" alt="MongoDB"/> <img src="https://img.shields.io/badge/LightRAG-purple" alt="LightRAG"/> </td> </tr> <tr> <td><b>📐 Embedding & Reranking</b></td> <td> <img src="https://img.shields.io/badge/SentenceTransformers-yellow" alt="SentenceTransformers"/> <img src="https://img.shields.io/badge/TEI-HuggingFace-FFD21E?logo=huggingface&logoColor=black" alt="TEI"/> <img src="https://img.shields.io/badge/CrossEncoder-ms--marco-blue" alt="CrossEncoder"/> <img src="https://img.shields.io/badge/rank--bm25-green" alt="BM25"/> </td> </tr> <tr> <td><b>⚡ Backend</b></td> <td> <img src="https://img.shields.io/badge/FastAPI-009688?logo=fastapi&logoColor=white" alt="FastAPI"/> <img src="https://img.shields.io/badge/Pydantic-E92063?logo=pydantic&logoColor=white" alt="Pydantic"/> <img src="https://img.shields.io/badge/Gunicorn-499848?logo=gunicorn&logoColor=white" alt="Gunicorn"/> <img src="https://img.shields.io/badge/Uvicorn-2E303E" alt="Uvicorn"/> <img src="https://img.shields.io/badge/JWT-black?logo=jsonwebtokens&logoColor=white" alt="JWT"/> </td> </tr> <tr> <td><b>📄 Document Processing</b></td> <td> <img src="https://img.shields.io/badge/Docling-IBM-054ADA?logo=ibm&logoColor=white" alt="Docling"/> <img src="https://img.shields.io/badge/Tesseract-OCR-blue" alt="Tesseract"/> <img src="https://img.shields.io/badge/PyMuPDF-PDF-red" alt="PyMuPDF"/> <img src="https://img.shields.io/badge/python--docx-DOCX-blue" alt="python-docx"/> <img src="https://img.shields.io/badge/Pillow-images-green" alt="Pillow"/> </td> </tr> <tr> <td><b>📡 Observability</b></td> <td> <img src="https://img.shields.io/badge/Langfuse-tracing-blue" alt="Langfuse"/> <img src="https://img.shields.io/badge/LangSmith-tracing-green?logo=langchain&logoColor=white" alt="LangSmith"/> <img src="https://img.shields.io/badge/Prometheus-E6522C?logo=prometheus&logoColor=white" alt="Prometheus"/> <img src="https://img.shields.io/badge/Grafana-F46800?logo=grafana&logoColor=white" alt="Grafana"/> <img src="https://img.shields.io/badge/Loki-2C3239?logo=grafana&logoColor=white" alt="Loki"/> </td> </tr> <tr> <td><b>📊 Evaluation & Testing</b></td> <td> <img src="https://img.shields.io/badge/RAGAS-evaluation-orange" alt="RAGAS"/> <img src="https://img.shields.io/badge/Pytest-0A9EDC?logo=pytest&logoColor=white" alt="Pytest"/> <img src="https://img.shields.io/badge/Ruff-D7FF64?logo=ruff&logoColor=black" alt="Ruff"/> <img src="https://img.shields.io/badge/Locust-load_test-green" alt="Locust"/> </td> </tr> <tr> <td><b>🐳 DevOps</b></td> <td> <img src="https://img.shields.io/badge/Docker-2496ED?logo=docker&logoColor=white" alt="Docker"/> <img src="https://img.shields.io/badge/GitHub_Actions-2088FF?logo=githubactions&logoColor=white" alt="GitHub Actions"/> <img src="https://img.shields.io/badge/NGINX-009639?logo=nginx&logoColor=white" alt="NGINX"/> <img src="https://img.shields.io/badge/uv-DE5FE9?logo=uv&logoColor=white" alt="uv"/> </td> </tr> </table> --- ## Progress ### Completed (42/46 Phases) | Category | Phases | Highlights | |----------|--------|------------| | **Foundation** | 0-4 | Clean architecture, LiteLLM async, multi-tenant RBAC | | **RAG Pipeline** | 5-6, 8-9, 18, 20, 22 | HyDE, CRAG, LangGraph agent, LightRAG, Docling, RAGAS | | **Infrastructure** | 10-14, 16-17, 19, 21 | vLLM (595ms), Docker (12 services), 6 LLM providers, Langfuse | | **Hardening** | 23-27b | Simplification refactor, JWT refresh, rate limit, MongoDB, CI/CD, GitFlow | | **Agentic** | 28A-28B, 29, 30-32b | Smart Router, Tool Use, MCP Integration, Context Engineering, mem0 Memory, Multi-Agent Orchestrator, Deep Lookup | | **Enterprise AOS** | 33-46 | Evaluation Gates, Task Queues, SurrealDB, GoF Patterns, Postgres Checkpointer, Background Daemons, Cancellation Pool | ### Current Stats | Metric | Value | |--------|-------| | Python files (src+tests) | 537 | | Lines of code in src | 61631 | | API endpoints | 97 | | Unit tests | 781 ✅ | | E2E tests | 29 ✅ | | Dependencies | 65 | | LLM providers | 6 | | Document formats | 10 | | Adapter packages | 9 (llm, embedding, vectordb, reranker, cache, stores, state, pubsub, queue) | --- ## Project Roadmap & Feature Checklist ### Foundation & Architecture - [x] **Phase 1**: Transition to Modular Monolith with Ports & Adapters separation - [x] **Phase 2**: Centralized dependency injection container (Composition Root in `deps.py`) - [x] **Phase 3**: JWT-based Authentication with refresh tokens - [x] **Phase 4**: Multi-Tenant Isolation (Organization → Department → User hierarchy) - [x] **Phase 41**: GoF Architecture Refactoring & GoF patterns alignment (technical debt resolution) - [x] **Phase 43**: V2 Core Architectural Readiness (monolith deconstruction and interface boundaries) ### Advanced RAG Pipeline - [x] **Phase 5**: Hybrid Retrieval (Lexical BM25 + Dense Semantic Vector Search) - [x] **Phase 6**: Reciprocal Rank Fusion (RRF) candidate merging - [x] **Phase 7**: Cross-Encoder Reranking (`ms-marco-MiniLM-L-6-v2`) run locally via SentenceTransformers - [x] **Phase 8**: Hypothetical Document Embeddings (HyDE) query expansion - [x] **Phase 9**: Corrective RAG (CRAG) chunk relevancy grader - [x] **Phase 10**: Graph RAG Integration (LightRAG entity-relation subgraphs) - [x] **Phase 18**: Layout-Aware Ingestion (IBM Docling table & structure extractor) - [x] **Phase 20**: Multi-Turn Query Rewriter & Conversation context integration - [x] **Phase 22**: Semantic Caching (cosine similarity $\ge 0.92$ isolated by tenant RBAC) - [x] **Phase 40**: SurrealDB Unified Store Adapter implementation ### Agentic & Memory Systems - [x] **Phase 28A**: Semantic Intent Router (Chitchat vs RAG vs Tool vs Multi-step agent) - [x] **Phase 28B**: Automatic Function-to-Tool Compiler & LangGraph Executor loops - [x] **Phase 29**: Model Context Protocol (MCP) server & client interface integration - [x] **Phase 30**: 5-Layer Context Engineering Engine (Lost-in-the-Middle mitigation) - [x] **Phase 31**: Multi-Tier Personalization Memory (Thread working memory + mem0 cloud semantic facts) - [x] **Phase 32**: Multi-Agent Orchestration (Planner + sub-agents + task folding + self-healing) - [x] **Phase 32b**: Deep Lookup Engine (Multi-hop search loops + gap analysis + scrapes + citations) - [x] **Phase 42**: Dynamic MCP Host & Self-Orchestrated Automation Engine (spawner, session pool, namespaced tool registry, scheduled jobs, approvals confirmation gates) - [x] **Phase 45**: Core Realignment, Continuous Empirical Learning Curator & Background Daemons ### Enterprise Governance & Security - [x] **Phase 23**: Microsoft Presidio PII Masking & redaction at the API gateway - [x] **Phase 24**: Input/Output Guardrails (prompt injection block & Vietnamese safety checkers) - [x] **Phase 25**: Complexity-Based Query Router (routing simple requests to local Ollama/Qwen) - [x] **Phase 26**: Daily USD token & cost budget limits per user - [x] **Phase 27**: KMS Envelope secrets encryption for databases & credentials - [x] **Phase 38**: Inclusion-Level Context Engineering (ContextBuilder) ### Infrastructure, State & Observability - [x] **Phase 11**: Production Observability (LangSmith & Langfuse trace span integration) - [x] **Phase 12**: Real-time Prometheus metrics endpoint (`/metrics`) & Grafana dashboard config - [x] **Phase 13**: Vision-LLM document content describer for scan/image files - [x] **Phase 16**: Custom vLLM serving container deployment configuration - [x] **Phase 17**: 12-Service Production Docker Compose stack with health checks - [x] **Phase 21**: GitHub Actions CI/CD pipeline (Ruff linting, formatting, security scanning, pytest) - [x] **Phase 33**: Evaluation-Driven CI/CD (eval gates + prompt versioning) - [x] **Phase 37**: DB-Centric Async Task Queue - [x] **Phase 44**: DeltaChannel Checkpointing & Postgres Checkpointer - [x] **Phase 46**: Infrastructure & Chat State Management Hub (Master Epic) ### React Admin Frontend & UI - [x] **Phase 35**: React Admin Frontend (Next.js dashboard, typewriter chat, waveform voice UI) ### Future Roadmap - [ ] **Phase 34**: Voice Pipeline (STT/TTS) — Real-time sentence-level audio pipelining: - Streaming STT with `faster-whisper` and `Silero-VAD` voice activity detection - Ultra-low latency TTS synthesis using `Kokoro` / `Orpheus` - Bidirectional audio streaming over WebSocket (`/api/v1/voice`) - Dynamic playback interruption on user speech detection - [ ] **Phase 36**: Terminal User Interface (TUI) for Local Execution - [ ] **Phase 39**: Conversational Audio Specialist & Podcast Pipeline --- ## License [MIT](LICENSE) — Copyright (c) 2025-2026 KnowledgeForge Contributors --- <p align="center"> _{Built with ❤️ using} <br/> <img src="https://img.shields.io/badge/FastAPI-009688?logo=fastapi&logoColor=white" alt="FastAPI"/> <img src="https://img.shields.io/badge/LangGraph-purple?logo=langchain&logoColor=white" alt="LangGraph"/> <img src="https://img.shields.io/badge/LiteLLM-black" alt="LiteLLM"/> <img src="https://img.shields.io/badge/mem0-blueviolet" alt="mem0"/> <img src="https://img.shields.io/badge/Qdrant-DC382D?logo=qdrant&logoColor=white" alt="Qdrant"/> <img src="https://img.shields.io/badge/RAGAS-orange" alt="RAGAS"/> <img src="https://img.shields.io/badge/Docker-2496ED?logo=docker&logoColor=white" alt="Docker"/> </p>