Content
# Vectorworks RAG + MCP
- Take in Vectorworks Python/VectorScript documents locally and enable cross-searching with FAISS.
- Provide `/search`, `/answer`, and `/get` endpoints with FastAPI and allow searching and source verification via a simple web UI.
- Simultaneously start an MCP server with WebSocket (JSON-RPC 2.0) and offer `vw.search`, `vw.answer`, and `vw.get` tools.
- Launch `app` (API/MCP) and `db` (Postgres 16) simultaneously with Docker Compose.
## Requirements
- Docker/Docker Compose (v2)
## Quick Start
1. Build
- `docker compose build`
2. Document Retrieval
- `docker compose run --rm app bash scripts/fetch_docs_minimal.sh`
- `docker compose run --rm -e GITHUB_TOKEN="$GITHUB_TOKEN" app bash scripts/fetch_github_vectorworks.sh`
3. Vector Generation (Embedding + FAISS)
- `docker compose run --rm app python -m app.indexer`
- If documents are added with an existing index, recreate it:
- `docker compose run --rm app python -m app.indexer --rebuild`
4. Launch (UI + MCP + Postgres)
- `docker compose up`
5. Access
- UI: `http://localhost:8000`
- MCP: `ws://localhost:8765`
Note
- Steps 2 and 3 can also be done with a one-liner:
- `docker compose run --rm app bash -lc 'scripts/fetch_docs_minimal.sh && python -m app.indexer'`
## Document Retrieval
The following command retrieves the minimum required documents (md/html only) to `data/`.
- Dependencies: `git`, `curl`
- Execution (inside the container):
- `docker compose run --rm app bash scripts/fetch_docs_minimal.sh`
- If a 403 error occurs due to network issues, the script skips the page and continues.
- If necessary, override the UA: `docker compose run --rm -e UA="Mozilla/5.0 ..." app bash scripts/fetch_docs_minimal.sh`
Retrieved Items (main points only)
- GitHub: Vectorworks/developer-scripting (Introduction/ Markdown)
- App Help: Scripting basic guidance (2022/2023/2024 key pages)
- Japanese site: VectorScript function index page + example pages
- Developer Wiki: VS Function Reference category index (HTML)
Note
- The indexer supports `.md`, `.markdown`, `.html`, `.htm`, and `.txt` only. PDFs are not included.
## GitHub (Vectorworks specified repositories only)
- Purpose: Clone (or update) only the following three repositories to `data/github/vectorworks/`.
- `Vectorworks/developer-scripting`
- `Vectorworks/developer-sdk`
- `Vectorworks/developer-worksheets`
- Execution (inside the container):
- `docker compose run --rm app bash scripts/fetch_github_vectorworks.sh`
- Destination: `data/github/vectorworks/<repo>`
- To add/change, specify with the `REPOS` environment variable (space or comma-separated).
- Example: `docker compose run --rm -e REPOS="Vectorworks/developer-scripting,Vectorworks/developer-sdk" app bash scripts/fetch_github_vectorworks.sh`
- Update method specification (optional): `UPDATE_MODE=pull` (default) or `UPDATE_MODE=reset`
- `pull`: Fast update with `git pull --ff-only --depth=1 --prune` (fails if local changes exist → fetch+reset for fail-safe)
- `reset`: Complete synchronization with `fetch --depth=1` followed by `reset --hard` (discards local changes)
Note
- The indexer targets text-based extensions (`.md`, `.html`, `.txt`, etc.) only.
## Vector Generation (Embedding + FAISS Index)
- Execution (inside the container):
- `docker compose run --rm app python -m app.indexer`
After execution, `index/` will contain `vw.faiss` and `meta.jsonl`.
(Make sure `data/` has documents.)
## Launch (UI + MCP + Postgres)
- `docker compose up --build`
- UI: `http://localhost:8000`
- MCP: `ws://localhost:8765`
- Postgres runs on the container's internal network (not exposed to the host)
## API Examples
- Search: `GET /search?q=PushAttrs&k=6`
- Example: `curl -s "http://localhost:8000/search?q=VectorScript&k=6" | jq .`
- Answer (draft): `GET /answer?q=...&k=6`
- Example: `curl -s "http://localhost:8000/answer?q=record+format" | jq .`
- Chunk retrieval: `GET /get?doc_id=...&chunk_id=...`
The UI is available at `GET /`, and you can check equivalent results from the search form.
## MCP (Model Context Protocol)
- Connect from VS Code: Add MCP with the following command
- `code --add-mcp '{"name":"vw_docs_local","url":"ws://localhost:8765"}'`
- Supported tools
- `vw.search({ query, k? })`
- `vw.answer({ query, k? })`
- `vw.get({ doc_id, chunk_id })`
Implementation is based on JSON-RPC 2.0 WebSocket server (`app/mcp_server.py`).
## Directory Structure
- `app/` Application main
- `api.py` FastAPI application
- `mcp_server.py` MCP (WebSocket) server
- `indexer.py` Document ingestion and vectorization (embedding + FAISS creation)
- `search.py` Core logic for search/answer
- `chunking.py` Chunking (aiming for about 700 token character length)
- `templates/` Web UI templates
- `data/` Original md/html (relative path is `doc_id`)
- `index/` FAISS and metadata (`vw.faiss`, `meta.jsonl`)
## Environment Variables (optional)
- `DATA_DIR` Data directory (default: `data`)
- `INDEX_DIR` Index output directory (default: `index`)
- `EMBED_MODEL` Embedding model (default: `sentence-transformers/all-MiniLM-L6-v2`)
- `CHUNK_CHARS` Chunk character length guideline (default: `2800` ≈ 700 tokens)
- `CHUNK_OVERLAP` Chunk overlap (default: `480`)
- `API_HOST` / `API_PORT` FastAPI bind (default: `0.0.0.0:8000`)
- `MCP_HOST` / `MCP_PORT` MCP bind (default: `0.0.0.0:8765`)
Postgres (for future use)
- `PGHOST` / `PGPORT` / `PGDATABASE` / `PGUSER` / `PGPASSWORD`
## Operation Notes
- If documents are updated, regenerate vectors: `python -m app.indexer`
- The initial run may take time due to downloading the embedding model.
- Since this runs on CPU, FAISS uses `IndexFlatIP` + normalized vectors (cosine equivalent).
## License / Notes
- This repository does not include documents. Place them in `data/` according to your usage policies and copyrights.
- `answer` is a draft based on excerpts. Verify the original text for final judgment.
