Content
# vision-mcp
A multimodal visual service based on MCP (Model Context Protocol) that supplements "vision" capabilities for coding models without native visual capabilities.
It connects to vision-enabled model vendors through **OpenAI SDK + OpenAI-compatible API**, converting screenshots, technical diagrams, charts, and video frames into high-quality text results for Cursor, Claude Code, Cline, VS Code Copilot, or other MCP clients to continue reasoning and coding.
## Feature Overview
Currently provides 8 MCP tools:
- `ui_to_artifact`
- Converts UI screenshots into code, design prompts, design specifications, or natural language descriptions
- `extract_text_from_screenshot`
- Extracts text from screenshots, suitable for code, terminal output, documents, and general text OCR
- `diagnose_error_screenshot`
- Analyzes error pop-ups, stack traces, and log screenshots, outputting positioning and repair suggestions
- `understand_technical_diagram`
- Interprets technical diagrams such as architecture diagrams, flowcharts, UML, ER diagrams, etc.
- `analyze_data_visualization`
- Reads charts/dashboard, distills trends, anomalies, and business highlights
- `ui_diff_check`
- Compares two UI screenshots, identifying visual differences and implementation deviations
- `image_analysis`
- General image understanding capability, suitable as a fallback when specialized tools are not covered
- `video_analysis`
- Supports local video analysis; when the provider declares support for native video, sends `video_url`; otherwise, automatically falls back to keyframe extraction
## Usage via npx (without cloning the repository)
vision-mcp has been published to npm, and you can run it with a single command using `npx`:
```bash
npx -y vision-mcp
```
Configure in the MCP client:
### VS Code Copilot
```json
{
"servers": {
"visionMcp": {
"type": "stdio",
"command": "npx",
"args": ["-y", "vision-mcp"]
}
},
"inputs": [
{
"type": "promptString",
"id": "api-key",
"description": "Vision API Key",
"password": true
},
{
"type": "promptString",
"id": "api-base-url",
"description": "API Base URL",
"password": false
},
{
"type": "promptString",
"id": "model",
"description": "Vision Model Name",
"password": false
}
]
}
```
Or configure in the project's `.vscode/mcp.json` and specify environment variables directly:
```json
{
"servers": {
"visionMcp": {
"type": "stdio",
"command": "npx",
"args": ["-y", "vision-mcp"],
"env": {
"VISION_API_BASE_URL": "https://dashscope.aliyuncs.com/compatible-mode/v1",
"VISION_API_KEY": "Your Key",
"VISION_MODEL": "qwen3.6-plus",
"VISION_SUPPORTS_NATIVE_VIDEO": "true"
}
}
}
}
```
### Cursor (`.cursor/mcp.json`)
```json
{
"mcpServers": {
"vision-mcp": {
"command": "npx",
"args": ["-y", "vision-mcp"],
"env": {
"VISION_API_BASE_URL": "https://dashscope.aliyuncs.com/compatible-mode/v1",
"VISION_API_KEY": "YOUR_API_KEY",
"VISION_MODEL": "qwen3.6-plus",
"VISION_SUPPORTS_NATIVE_VIDEO": "true"
}
}
}
}
```
### Claude Code
```json
{
"mcpServers": {
"vision-mcp": {
"command": "npx",
"args": ["-y", "vision-mcp"],
"env": {
"VISION_API_BASE_URL": "https://dashscope.aliyuncs.com/compatible-mode/v1",
"VISION_API_KEY": "YOUR_API_KEY",
"VISION_MODEL": "qwen3.6-plus"
}
}
}
}
```
### Cline
```json
{
"mcpServers": {
"vision-mcp": {
"command": "npx",
"args": ["-y", "vision-mcp"],
"env": {
"VISION_API_BASE_URL": "https://dashscope.aliyuncs.com/compatible-mode/v1",
"VISION_API_KEY": "YOUR_API_KEY",
"VISION_MODEL": "qwen3.6-plus"
}
}
}
}
```
> **Note for Windows users**: If the MCP client cannot correctly parse the `npx` command, use:
> ```json
> {
> "command": "cmd",
> "args": ["/c", "npx", "-y", "vision-mcp"]
> }
> ```
## Local Development Mode
```bash
npm install
npm run build
npm start
```
## Configuration
Copy `.env.example` to `.env`:
```bash
cp .env.example .env
```
### Required Configuration
```env
VISION_API_BASE_URL=
VISION_API_KEY=
VISION_MODEL=
```
### Optional Configuration
```env
VISION_SUPPORTS_NATIVE_VIDEO=false
VISION_NATIVE_VIDEO_FPS=2
VISION_MAX_IMAGES_PER_REQUEST=10
VISION_IMAGE_DETAIL=auto
VISION_TIMEOUT=60000
```
### Configuration Description
| Variable | Required | Default Value | Description |
|---|---|---:|---|
| `VISION_API_BASE_URL` | Yes | - | OpenAI-compatible API base URL |
| `VISION_API_KEY` | Yes | - | API Key |
| `VISION_MODEL` | Yes | - | Model Name |
| `VISION_SUPPORTS_NATIVE_VIDEO` | No | `false` | Whether the provider supports native video input |
| `VISION_NATIVE_VIDEO_FPS` | No | `2` | FPS hint for native video input, range `0.1` - `10` |
| `VISION_MAX_IMAGES_PER_REQUEST` | No | `10` | Maximum number of images per request |
| `VISION_IMAGE_DETAIL` | No | `auto` | Image detail: `low` / `high` / `auto` |
| `VISION_TIMEOUT` | No | `60000` | Request timeout (ms) |
## Requirements
- Node.js >= 18
- For `video_analysis` keyframe extraction, ensure the system has:
- `ffmpeg`
- `ffprobe`
## Tool Input Summary
| Tool | Main Parameters |
|---|---|
| `ui_to_artifact` | `image`, `output_format?`, `extra_prompt?` |
| `extract_text_from_screenshot` | `image` |
| `diagnose_error_screenshot` | `image` |
| `understand_technical_diagram` | `image`, `diagram_type?` |
| `analyze_data_visualization` | `image` |
| `ui_diff_check` | `image_before`, `image_after`, `description?` |
| `image_analysis` | `image`, `prompt?` |
| `video_analysis` | `video`, `prompt?` |
## Video Capability Description
- When `VISION_SUPPORTS_NATIVE_VIDEO=false`: Extract keyframes from local video (first/middle/last) and send keyframes as multiple images to the visual model for analysis
- When `VISION_SUPPORTS_NATIVE_VIDEO=true`: Directly encode local video into a Base64 Data URL with MIME and send `video_url` according to the provider's extension format
- Current native video branch is prepared for OpenAI-compatible provider extensions (e.g., DashScope/Bailian's `video_url`) and does not represent standard OpenAI official schema native support for video
## Usage Tips in VS Code Copilot
When using Copilot Chat + vision-mcp with non-visual models:
1. **Do not paste images in Chat** (models cannot see them)
2. Save images to a local directory
3. Explicitly call tools with **file paths** in Chat:
```text
Please use the extract_text_from_screenshot tool to process C:/path/to/screenshot.png
```
Copilot will pass the file path to vision-mcp, which will read the file and call the visual model.
## Development
```bash
npm run build
npm test
```
## Troubleshooting
### Error: Missing environment variables at startup
Check for the presence of:
- `VISION_API_BASE_URL`
- `VISION_API_KEY`
- `VISION_MODEL`
### Video analysis fails, citing missing ffmpeg
Install `ffmpeg` / `ffprobe` and ensure they are in the system PATH.
### Model request timeout
- Increase `VISION_TIMEOUT`
- Reduce the number of images or resolution
- Check model vendor interface stability
### Authentication error returned
- Check `VISION_API_KEY`
- Check if `VISION_API_BASE_URL` matches the vendor's requirements
- Check if `VISION_MODEL` exists and has visual capabilities
### Image file does not exist or path is incorrect
vision-mcp accepts relative paths (relative to the current working directory) and absolute paths. Ensure the path is correct and use absolute paths if possible.
## License
MIT
