mindvault

# Mindvault **Status: Phase 2. Retrieval pipeline validated at F1 0.728 on 827-memory synthetic benchmark. REST API + MCP operational. Prompt-engineered v6 filter deployed on Qwen3 14B AWQ via vLLM.** Mindvault is an AI memory management system for general-purpose agents: personal assistants, business operations agents, and similar applications. It combines PostgreSQL full-text search, pgvector semantic search, and an LLM relevance filter to achieve F1 0.728 on a 827-memory benchmark, outperforming both pure vector search and cross-encoder reranking approaches. ## The Hypothesis Current memory systems for AI agents rely on vector similarity search (they find things that *look similar* to the query). Mindvault tests whether PostgreSQL full-text search combined with a focused LLM that reasons over candidates, filtering false positives and assembling coherent responses, produces better retrieval quality at acceptable latency and cost. **Result so far:** On synthetic data (827 memories, 98 queries), the v6 prompt-engineered pipeline achieves F1 0.728, a +37% improvement over the initial hybrid reranker approach (0.531) and +16% over the same model with a flat prompt (0.629). The improvement came entirely from prompt engineering, not infrastructure changes. See [Benchmark Reports](#benchmark-reports) for the full story. ## How It Works ``` Agent -> "What are the payment terms for our logistics suppliers?" | (REST API or MCP tool call) Memory Service -> Hybrid retrieval (FTS + pgvector cosine) | (merged via Reciprocal Rank Fusion) Local Memory LLM -> evaluates candidates, selects Gold, synthesizes answer | (v6 balanced filter prompt with global reasoning) Agent <- gets exactly what it needs ``` The agent never touches memory storage directly. It sends natural language questions via REST API or MCP tool interface. The service searches, filters via LLM, and returns a curated answer with source references. ## Key Design Choices - **PostgreSQL** for storage, OR-based full-text search, and pgvector cosine similarity - **Tiered abstraction (L0/L1/L2):** L0 search abstract (~76 chars), L1 structured overview (~221 chars), L2 full source content (~2900 chars). LLM filter sees L1; L2 used only for cross-encoder scoring if enabled - **Hybrid retrieval:** FTS keyword matching + pgvector semantic search, merged via Reciprocal Rank Fusion (RRF). No score normalisation needed - **Integer index mapping:** candidates presented as `[0]`, `[1]`, `[2]` instead of UUIDs. Eliminates UUID hallucination (10-15 tokens of random hex per pick) - **v6 "balanced filter" prompt:** single global evaluation sentence before selection. "Select ALL candidates that provide a unique piece of the puzzle." Achieves both high recall (0.917) and precision at ~4s latency - **Local LLM:** Qwen3 14B AWQ served via vLLM on RTX 5070 Ti 16GB. The `/no_think` suffix suppresses chain-of-thought reasoning for JSON-only output - **Configurable retrieval:** single-pass (filter + synthesize from L1) or two-pass (filter L1, then synthesize from full content) - **Full operation logging:** every LLM call logged with prompts, responses, candidates, and selections - **Separate skills storage:** executable instructions with embedded scripts, searched independently - **MCP + REST:** dual interface with MCP tools for agent integration and REST API for direct HTTP access - **OpenAI-compatible LLM interface:** works with vLLM, llama.cpp, Ollama, or any compatible API ## What's Implemented **Full CRUD for memories and skills** via REST API, MCP tools, and CLI: | Operation | Memory | Skill | |-----------|--------|-------| | Create | `POST /memory/store` | `POST /skill/store` | | Read | `GET /memory/{id}`, `POST /memory/list` | `GET /skill/get/{name}`, `POST /skill/find` | | Update | `PATCH /memory/{id}` | `PATCH /skill/update/{name}` | | Delete | `DELETE /memory/{id}` | `DELETE /skill/delete/{name}` | | Import file | `POST /memory/import` | `POST /skill/import` | | Search | `POST /memory/retrieve` (LLM-filtered) | `POST /skill/find` (FTS) | All REST endpoints except file import have matching MCP tools. File import is REST-only (multipart upload); agents using MCP call `memory_store` / `skill_store` directly with the text content. MCP is available via Streamable HTTP (mounted at `/mcp`) or stdio (`uv run mindvault-mcp`). When `MINDVAULT_API_KEY` is set, MCP over HTTP requires the same Bearer token as REST. See [API Reference](docs/api-reference.md) for full endpoint documentation with examples. ### CLI ```bash mindvault memory list mindvault memory get <id> --output file.md mindvault memory store --file notes.md --category domain_knowledge mindvault memory update <id> --file updated.md mindvault memory delete <id> mindvault skill get deploy-procedure --output deploy.md mindvault skill store --file runbook.md ``` See [CLI Reference](docs/cli.md) for all commands. ### Infrastructure - Full write and retrieval logging to PostgreSQL - 281 automated tests (pytest + pytest-asyncio) - Service layer shared across REST, MCP, and CLI interfaces - Benchmark framework with multi-approach comparison, threshold sweeps, multi-run averaging, and per-difficulty breakdowns - Synthetic datasets at two scales: `synthetic_v1` (50 memories) and `synthetic_v2` (827 memories with procedurally generated lookalike distractors) ## Current Performance Measured on `synthetic_v2_gold` (827 memories, 98 queries) with Qwen3 14B AWQ via vLLM on RTX 5070 Ti 16GB: | Approach | Precision | Recall | F1 | Latency p50 | |---|---|---|---|---| | Baseline (FTS top-5, no LLM) | 0.176 | 0.783 | 0.283 | 6ms | | **v6 single-pass (production)** | **0.584** | **0.917** | **0.728** | **~4s** | By difficulty: | Difficulty | Baseline F1 | v6 Single-pass F1 | |---|---|---| | Easy (33 queries) | 0.333 | 0.858 | | Medium (37 queries) | 0.250 | 0.775 | | Hard (28 queries) | 0.267 | 0.512 | ## What's Next - **Purpose-trained filter model.** The current pipeline uses a general-purpose 14B model prompted for relevance filtering. A small (1-3B) model fine-tuned specifically on the retrieval filter task could match or exceed the v6 F1 while cutting latency from ~4s back to sub-second. Every LLM call is logged with prompts, responses, and gold labels, providing training data for this. - **Hard query improvement.** The v6 prompt plateaus at 0.512 F1 on Hard queries (broad, multi-memory answers). Tested and rejected: L2 peeking, cross-encoder reranking. Next candidate: metadata injection (dates, tags) into L1 summaries to help the LLM distinguish Gold from similar-but-stale distractors. - **Real-corpus validation.** All current benchmarks use synthetic data. A real-world memory export is the next stress test for the pipeline. - Additional test coverage for MCP tools and edge cases. ## Tech Stack | Component | Technology | |-----------|-----------| | Language | Python | | Package Manager | uv (Astral) | | Web Framework | FastAPI | | MCP SDK | mcp (FastMCP) | | Database | PostgreSQL (pgvector enabled) | | Database Driver | SQLAlchemy async engine + asyncpg | | LLM | Qwen3 14B AWQ (production); llama3.2:3b (original baseline) | | LLM Runtime | vLLM (production); Ollama / llama.cpp (compatible) | | Embeddings | BAAI/bge-small-en-v1.5 (384-dim) via sentence-transformers | | Reranker | BAAI/bge-reranker-v2-m3 (available but disabled; see [report](benchmarks/reports/2026-04-10_prompt-engineering-v4-to-v8.md)) | | Tests | pytest + pytest-asyncio + httpx | ### Hardware (benchmark reference) All latency numbers in this README and the benchmark reports were measured on: - **GPU:** NVIDIA RTX 5070 Ti 16GB - **OS:** WSL2 (Linux 6.6.87.2-microsoft-standard-WSL2) - **LLM serving:** vLLM in Docker, `max_model_len=16384` - **Model:** Qwen3 14B AWQ (4-bit quantized) ## Quick Start ```bash uv sync # install dependencies cp .env.example .env # configure (edit with your DB/LLM settings) uv run uvicorn mindvault.main:app --host 127.0.0.1 --port 8080 --reload ``` See [Setup Guide](docs/setup.md) for full installation instructions including database setup and LLM configuration. ### Security Mindvault supports API key authentication. Generate a key and add it to `.env`: ```bash mindvault-keygen --env >> .env ``` When set, all memory and skill endpoints require `Authorization: Bearer <key>`. Health endpoints remain open. If unset, the API is open (for local development). **Important:** API keys sent over plain HTTP are visible to anyone on the network path. Always use HTTPS when the server is reachable from another machine. Generate a dev certificate with `mindvault-gencert`, or use a reverse proxy with a real CA certificate for production. ```bash # Quick HTTPS setup for development mindvault-gencert uv run uvicorn mindvault.main:app \ --ssl-certfile=cert.pem --ssl-keyfile=key.pem \ --host 127.0.0.1 --port 8443 ``` See [Deployment Guide](docs/deployment.md) for production TLS setup (Caddy, nginx, Traefik, Cloudflare Tunnel, Tailscale Funnel). A future version will support per-namespace keys for multi-agent deployments. See [Authentication](docs/authentication.md) for details. ### Using the CLI ```bash # Store a document as a memory mindvault memory store --file contract.md --category domain_knowledge # Ask a question curl -X POST http://localhost:8080/memory/retrieve \ -H "Content-Type: application/json" \ -d '{"question": "Who is our DHL account representative?"}' # Browse and export mindvault memory list mindvault memory get <id> --output exported.md ``` ## Benchmarking The benchmark framework compares retrieval quality across multiple approaches: 1. **Baseline:** PostgreSQL full-text search, top-N by rank, no LLM 2. **Mindvault single-pass:** Hybrid FTS + vector, RRF merge, LLM filter/synthesize from L1 3. **Mindvault two-pass:** Same retrieval, LLM filter from L1 + synthesize from full content Metrics: precision, recall, F1, token usage, latency (p50/p95). Results broken down by query difficulty (easy/medium/hard). Multi-run averaging via `--runs N` reports F1 +/- stddev. ```bash # Run the production configuration benchmark PG_DB=mindvault_test uv run python benchmarks/run_benchmark.py \ benchmarks/datasets/synthetic_v2_gold.json # Run baseline only (no LLM needed) PG_DB=mindvault_test uv run python benchmarks/run_benchmark.py \ benchmarks/datasets/synthetic_v2_gold.json --baseline-only ``` Custom datasets can be added to `benchmarks/datasets/custom/`. Result JSONs land in `benchmarks/results/` (gitignored). ## Benchmark Reports Experiment-by-experiment progress on the retrieval pipeline lives in [`benchmarks/reports/`](benchmarks/reports/README.md). Each report is self-contained: hardware, dataset, method, results, findings, and reproducibility commands. Reports are chronological; read top-to-bottom to follow the story. The current production configuration and headline numbers are documented in [`benchmarks/reports/README.md`](benchmarks/reports/README.md). Highlights: - [2026-04-06 Small Model Shootout](benchmarks/reports/2026-04-06_small-model-shootout.md): picked llama3.2:3b as the initial filter model - [2026-04-07 Hybrid Routing](benchmarks/reports/2026-04-07_hybrid-routing.md): introduced hybrid retrieval, validated 0.55/0.55 operating point - [2026-04-08 Hybrid Scale Validation](benchmarks/reports/2026-04-08_hybrid-scale-validation.md): re-validated at 16x scale on synthetic_v2 (827 memories) - [2026-04-08 FTS Top-K Sweep](benchmarks/reports/2026-04-08_fts-top-k-sweep.md): discrimination > recall, smaller candidate sets improve F1 - [**2026-04-10 Prompt Engineering v4 to v8**](benchmarks/reports/2026-04-10_prompt-engineering-v4-to-v8.md): the breakthrough. v6 prompt + Qwen3 14B AWQ delivers F1 0.728 (+37% over prior best). Tested and rejected: per-candidate CoT, L2 peeking, cross-encoder reranking ## Testing Tests run against a `mindvault_test` PostgreSQL database. LLM calls are mocked for speed and determinism. ```bash # Run all tests PG_DB=mindvault_test uv run pytest -v # Run a specific test file PG_DB=mindvault_test uv run pytest tests/test_memory_store.py -v ``` See `plans/` for architecture documentation and design decisions. ## License Apache License 2.0. See [LICENSE](LICENSE) for details.