cobo

# Japanese Vector Search System This is a Japanese vector search system using sqlite-vec. ## Features - Fast Search: Optimized vector search with sqlite-vec - Japanese Support: Utilizes the pfnet/plamo-embedding-1b model - Lightweight: SQLite-based with minimal dependencies - MCP Support: Model Context Protocol tool server - Performance Optimization: Caching, connection pooling, PRAGMA optimizations ## Project Structure ``` . ├── lib/ # Core libraries │ ├── vector_utils.py # Vector search utilities │ └── data_processing.py # Data processing and DB construction ├── test_search.py # Simple search test ├── benchmark.py # Performance measurement ├── build_db.py # Database construction script ├── server.py # MCP server ├── srv.sh # Server startup script └── README.md ``` ## Setup ### 1. Install Dependencies ```bash uv sync ``` ### 2. Build the Database ```bash uv run build_db.py ``` ### 3. Verify Operation ```bash uv run test_search.py info ``` ## Usage ### Basic Search Test ```bash # Display database information uv run test_search.py info # Single search uv run test_search.py "大学" # Interactive mode uv run test_search.py ``` ### Performance Measurement ```bash # Basic benchmark uv run benchmark.py # Detailed analysis uv run benchmark.py --detailed # Custom queries uv run benchmark.py --queries "大学" "授業" "履修" ``` ### MCP Server ```bash # Start the server ./srv.sh start # Check server status ./srv.sh status # Stop the server ./srv.sh stop ``` ## Performance Optimization ### Implemented Optimizations - Embedding Cache: Speeds up identical queries - Connection Pool: Reuses database connections - PRAGMA Optimizations: Optimizes SQLite settings - WAL mode - Large cache (20,000 pages) - Memory temporary storage - 4KB page size - 256MB mmap ### Performance Results - Initial search: ~0.1 seconds - Cache hit: ~0.05 seconds - DB search: ~0.007 seconds ## Configuration ### Data Source Configuration Configurable in the `.env` file: ```bash # Use FTP source USE_FTP_SOURCE=true FTP_HOST=192.168.7.48 FTP_USER=anonymous FTP_PASS= FTP_DATA_DIR=/data # Use local source USE_FTP_SOURCE=false LOCAL_DIR=./data ``` ### Server Configuration Configurable in the `.server_config` file: ```bash TRANSPORT=streamable-http PORT=8080 HOST=0.0.0.0 STATELESS=false ``` ## System Requirements - Python 3.12 or higher - At least 2GB of RAM (for model loading) - Apple Silicon recommended (MPS support) ## Technical Details ### Technologies Used - Vector DB: sqlite-vec v0.1.6 - Embedding Model: pfnet/plamo-embedding-1b (2048 dimensions) - MCP Framework: FastMCP - Language: Python 3.12 ### Architecture 1. Data Processing: Parsing and chunking Markdown files 2. Vectorization: Generating embeddings for Japanese text 3. Indexing: Fast search indexing with sqlite-vec 4. Search: Retrieving similar documents using KNN search ## Troubleshooting ### Common Issues 1. Model loading error ```bash # Clear cache rm -rf ~/.cache/huggingface/ ``` 2. Database error ```bash # Rebuild DB rm search.db* uv run python build_db.py ``` 3. Performance issues ```bash # Run detailed analysis uv run benchmark.py --detailed ``` ## License MIT License