MCP-knowledgegraph

# Repository to Knowledge Graph (repo-to-kg) This project is a benchmarking and implementation platform for converting codebase repositories into Knowledge Graphs (KGs) using Model Context Protocol (MCP) and SQLite. ## Directory Structure - **`src/`**: Core implementation of the knowledge graph query layer, embeddings index, and GraphRAG pipeline. - `query/graph_queries.py`: Relational SQLite parameterized search,caller/callee tracing, transitive import tracking, impact analysis, and BFS pathfinding. - `embeddings/embedder.py`: Signature/docstring extractor, batch encoding using local `sentence-transformers`, and SQLite vector search. - `rag/graph_rag.py`: Pluggable GraphRAG answering pipeline supporting Mock, Gemini, and OpenAI LLM providers. - `api/main.py`: FastAPI server exposing all graph search and GraphRAG endpoints. - `cli.py`: Command-line tool to run search and GraphRAG queries. - **`benchmarks/`**: Resources, questions, and automated scripts for benchmarking knowledge graph extraction capabilities. - **`supabase/`**: Cloned repository used as the initial benchmark target. The graph is stored in `supabase/.repo-to-kg/db.sqlite`. - **`tests/`**: Unit tests (`pytest`) and end-to-end smoke tests verifying the GraphRAG pipeline. --- ## Setup & Installation 1. **Virtual Environment**: Ensure you are using Python 3.11. Create and activate a virtual environment: ```bash python3.11 -m venv .venv source .venv/bin/activate ``` 2. **Install Dependencies**: ```bash pip install -r requirements.txt ``` 3. **Build Embeddings Index**: Day 4 builds on top of the Day 1-3 parsed SQLite graph. To compute and save embeddings for all Functions, Classes, and Methods: ```bash PYTHONPATH=. python -m src.cli index ``` This model downloads the lightweight `all-MiniLM-L6-v2` transformer model locally (stored in `.cache/`) and creates/populates the `symbol_embeddings` table. --- ## Running the API Server Start the FastAPI application on port 8000: ```bash PYTHONPATH=. uvicorn src.api.main:app --reload ``` --- ## Example Queries (curl & API) ### 1. Fetch Entity Details Fetches a node matching a name (file or symbol) and all of its direct incoming/outgoing relationships. ```bash curl http://localhost:8000/entity/getOrgAIDetails ``` ### 2. Impact Analysis Lists all upstream callers and dependent files (transitive incoming edges) up to a depth of 3: ```bash curl http://localhost:8000/impact/getOrgAIDetails?depth=3 ``` ### 3. Semantic Search Performs vector similarity search over codebase symbols: ```bash curl -X POST -H "Content-Type: application/json" \ -d '{"query": "where is organization AI details retrieved", "k": 3}' \ http://localhost:8000/search ``` ### 4. GraphRAG Question-Answering Ask natural language questions. It retrieves the seed symbols via embeddings, expands 1-2 hops for context (callers, callees, containing files, siblings), and queries the LLM: ```bash curl -X POST -H "Content-Type: application/json" \ -d '{"question": "How is organization AI details determined?"}' \ http://localhost:8000/ask ``` *(Note: If `GEMINI_API_KEY` or `OPENAI_API_KEY` are not set in the environment, this automatically falls back to a simulated `MockProvider` response citing correct source code blocks).* --- ## Running Git History Ingestion To parse and index the Git history of the codebase subtree and attribute line-level authorship to symbols: ```bash cd repo-to-kg pnpm install pnpm build node bin/cli.js git-index ../supabase ``` --- ## Why-Layer API & CLI Day 5 connects code symbols to their git history, author ownership, and squash-merge PR/issue references. ### Example Queries (curl & API) #### 1. Why-Layer Evidence Bundle (Killer Curl Demo) Fetches a symbol's entire history timeline, author line ownership breakdown, and its file churn statistics. ```bash curl http://localhost:8000/why/getOrgAIDetails ``` #### 2. Symbol History Retrieves all commits touching a symbol ordered chronologically by committed date. ```bash curl http://localhost:8000/history/getOrgAIDetails ``` #### 3. Churn & Hotspots Retrieves the top N most frequently modified files and symbols in the index. ```bash curl http://localhost:8000/hotspots?limit=5 ``` --- ## CLI Usage For quick terminal-based queries: - **Ask Question (GraphRAG)**: ```bash PYTHONPATH=. python src/cli.py ask "How is organization AI details determined?" ``` - **Semantic Search**: ```bash PYTHONPATH=. python src/cli.py search "where is checkEntitlement used" -k 3 ``` - **Why Evidence Bundle**: ```bash PYTHONPATH=. python src/cli.py why getOrgAIDetails ``` - **Symbol Commit History**: ```bash PYTHONPATH=. python src/cli.py history getOrgAIDetails ``` - **Churn Hotspots**: ```bash PYTHONPATH=. python src/cli.py hotspots --limit 5 ``` --- ## Testing & Validation ### Sanity Check Check graph node/edge statistics and verify that all target symbols have generated embeddings: ```bash PYTHONPATH=. python src/utils/sanity_check.py ``` ### Run Unit Tests & Smoke Tests Run unit tests checking path-finding, dependency calculation, caller/callee tracing, and GraphRAG flow: ```bash PYTHONPATH=. pytest tests/test_queries.py PYTHONPATH=. python tests/smoke_test.py PYTHONPATH=. python tests/test_api.py ```