Content
# GlassCloud
Cloud server for Rokid Bridge and GlassBridge — handles device linking, MCP tool relay, image storage with gallery, and server-side LLM scene description.
## What It Does
GlassCloud is the backend for Rokid smart glasses apps. It provides:
1. **Device Linking** — QR code or token-based pairing between the web console and the Android app
2. **MCP Tool Relay** — Proxies Gmail and Calendar tool calls from the glasses to Google APIs via WebSocket
3. **Image Gallery** — Receives scene images uploaded from the glasses, stores them with metadata, and displays them in a web gallery
4. **Server-Side LLM** — When an image arrives without a description, automatically calls Gemini or ChatGPT to generate one, then returns it to the app for display on the glasses
5. **Public Gallery Links** — Per-user public share tokens so galleries can be viewed without logging in
```
┌─────────────────┐ ┌─────────────────┐ ┌─────────────────┐
│ Rokid Glasses │────▶│ GlassCloud │────▶│ Google APIs │
│ + Android App │ WS │ (This Server) │ │ Gmail/Calendar │
└─────────────────┘ └─────────────────┘ └─────────────────┘
│ │
REST API │ │ Server-side LLM
▼ ▼
┌─────────────────┐
│ Web Console │
│ Gallery · LLM │
│ Settings · QR │
└─────────────────┘
```
---
## Quick Start
```bash
# Install dependencies
npm install
# Copy and fill in environment variables
cp .env.example .env
# Edit .env — see Configuration section below
# Development (auto-reload)
npm run dev
# Production build + run
npm run build
npm start
```
Open **http://localhost:3000/console** to access the web console.
---
## Configuration
All environment variables are validated at startup with Zod. The server will print the missing fields and exit if required variables are absent.
### Generating Secrets
```bash
node -e "console.log(require('crypto').randomBytes(32).toString('base64'))"
```
Run this twice — once for `JWT_SECRET`, once for `ENCRYPTION_KEY`.
### Required Variables
| Variable | Description |
|---|---|
| `JWT_SECRET` | Signs device JWTs. Min 32 chars. Must be random and secret. |
| `ENCRYPTION_KEY` | Encrypts OAuth tokens and LLM API keys at rest (AES-256-GCM). Min 32 chars. |
| `GOOGLE_CLIENT_ID` | Google OAuth 2.0 client ID |
| `GOOGLE_CLIENT_SECRET` | Google OAuth 2.0 client secret |
| `GOOGLE_CALLBACK_URL` | OAuth redirect URI — must match what's registered in Google Cloud Console (e.g. `https://yourserver.com/auth/google/callback`) |
### Optional Variables
| Variable | Default | Description |
|---|---|---|
| `PORT` | `3000` | HTTP server port |
| `NODE_ENV` | `development` | `development` \| `production` \| `test` |
| `LOG_LEVEL` | `info` | `trace` \| `debug` \| `info` \| `warn` \| `error` \| `fatal` |
| `DATABASE_PATH` | `./data/glasscloud.db` | SQLite database file path |
| `CORS_ORIGINS` | `http://localhost:3000` | Comma-separated list of allowed CORS origins |
| `RATE_LIMIT_WINDOW_MS` | `60000` | Rate limit window in milliseconds |
| `RATE_LIMIT_MAX_REQUESTS` | `60` | Max requests per window per IP (applies to `/api/*`) |
| `WS_PING_INTERVAL_MS` | `30000` | WebSocket keepalive ping interval |
| `WS_PONG_TIMEOUT_MS` | `10000` | Time to wait for pong before closing connection |
### Example `.env`
```bash
PORT=3000
NODE_ENV=production
LOG_LEVEL=info
DATABASE_PATH=./data/glasscloud.db
JWT_SECRET=<generate with command above>
ENCRYPTION_KEY=<generate with command above>
CORS_ORIGINS=https://yourserver.com
GOOGLE_CLIENT_ID=123456789.apps.googleusercontent.com
GOOGLE_CLIENT_SECRET=GOCSPX-xxxxx
GOOGLE_CALLBACK_URL=https://yourserver.com/auth/google/callback
```
### Google OAuth Setup
1. Go to [Google Cloud Console → APIs & Credentials](https://console.cloud.google.com/apis/credentials)
2. Create an **OAuth 2.0 Client ID** (Web application)
3. Add your callback URL to **Authorized redirect URIs**: `https://yourserver.com/auth/google/callback`
4. Enable the **Gmail API** and **Google Calendar API** in the project
5. Add test users under **OAuth consent screen → Test users** (required while app is in Testing mode)
> **Note on Google Scopes:** This app requests `gmail.readonly` and `calendar.events`. For public deployment with more than 100 users, Google requires a CASA security assessment. For private or testing use, keep the app in Testing mode.
---
## LLM Settings (Server-Side Scene Description)
GlassCloud can automatically generate scene descriptions for images uploaded without one (e.g. when the Rokid app is in "Bypass LLM" mode). The generated description is:
- Saved to the database and shown in the gallery
- Returned in the upload API response so the app can display it on the glasses immediately
### Configuration
1. Sign in at `/console` and click **LLM Settings →** in the header
2. Select a provider (**Gemini** or **ChatGPT**) and paste your API key
3. Optionally add custom prompts and select one as active
The default prompt is: *"Describe the scene in this image concisely."*
### API Keys
| Provider | Where to get it |
|---|---|
| **Gemini** | [Google AI Studio → API keys](https://aistudio.google.com/app/apikey) |
| **ChatGPT** | [OpenAI Platform → API keys](https://platform.openai.com/api-keys) |
API keys are encrypted at rest using `ENCRYPTION_KEY` (AES-256-GCM). If no API key is configured, server-side LLM is silently skipped and images with empty descriptions are stored as-is.
---
## Web Console Pages
| URL | Description |
|---|---|
| `/console` | Dashboard: stats, device linking, MCP service management |
| `/console/link` | Generate QR code / link token for pairing a device |
| `/console/gallery` | Image gallery (table view with thumbnails, details, delete) |
| `/console/gallery/photo/:id` | Full-size image with metadata |
| `/console/settings/llm` | LLM provider, API key, and prompt management |
| `/public/gallery/:token` | Public gallery (no login required) |
| `/public/gallery/:token/photo/:id` | Public photo details |
---
## REST API Endpoints
### Device & Auth
| Endpoint | Method | Auth | Description |
|---|---|---|---|
| `/health` | GET | None | Health check with connection stats |
| `/auth/google` | GET | None | Initiate Google OAuth flow |
| `/auth/google/callback` | GET | None | OAuth callback |
| `/api/link/exchange` | POST | None | Exchange link token for device JWT |
| `/api/devices` | GET | Device JWT | List linked devices |
| `/api/devices/:id` | DELETE | Session | Unlink a device |
### Images
| Endpoint | Method | Auth | Description |
|---|---|---|---|
| `/api/images/upload` | POST | Device JWT | Upload image + metadata (multipart/form-data) |
| `/api/images/:id` | GET | Device JWT | Get image metadata by ID |
#### Upload fields
| Field | Type | Required | Description |
|---|---|---|---|
| `image` | file | Yes | JPEG, PNG, or WebP — max 20 MB |
| `scene_description` | string | Yes | Scene description (may be empty string) |
| `device_model` | string | Yes | Device model name |
| `device_instance_id` | string | Yes | Unique device identifier |
| `captured_at` | string | No | ISO 8601 timestamp when image was captured |
| `image_width` | number | No | Image width in pixels |
| `image_height` | number | No | Image height in pixels |
| `llm_provider` | string | No | LLM used on-device (e.g. `Gemini`, `ChatGPT`, `None`) |
If `scene_description` is empty and a server-side LLM is configured, the server will call the LLM and include `llmDescription` in the response:
```json
{
"imageId": "uuid",
"originalFilename": "uuid.jpg",
"thumbnailFilename": "uuid_thumb.jpg",
"receivedAt": 1714000000000,
"capturedAt": 1714000000000,
"llmDescription": "A person walking through a crowded market..."
}
```
### MCP
| Endpoint | Method | Auth | Description |
|---|---|---|---|
| `/api/mcp/services` | GET | Device JWT | List available MCP services and tools |
---
## WebSocket Protocol
Connect at: `ws://localhost:3000/ws?deviceId=YOUR_DEVICE_ID`
### Client → Server
```json
// Execute a tool
{ "type": "tool_execute", "requestId": "uuid", "serverId": "gmail",
"toolName": "gmail.get_unread", "arguments": { "maxResults": 10 } }
// Link device to user account
{ "type": "link_device", "requestId": "uuid",
"linkToken": "from-qr-code", "deviceId": "..." }
// Get user account info
{ "type": "get_user_account", "requestId": "uuid", "deviceId": "..." }
```
### Server → Client
```json
// Immediate progress (for voice feedback while tool runs)
{ "type": "tool_progress", "requestId": "uuid",
"status": "executing", "message": "Checking your emails..." }
// Tool result
{ "type": "tool_result", "requestId": "uuid",
"result": { "success": true, "content": "You have 3 unread emails..." } }
// Error
{ "type": "error", "requestId": "uuid", "error": "Token expired" }
```
---
## Available MCP Tools
### Gmail
| Tool | Description |
|---|---|
| `gmail.get_unread` | Get unread email count and summaries |
| `gmail.search` | Search emails by query |
| `gmail.get_message` | Get full email content by ID |
### Calendar
| Tool | Description |
|---|---|
| `calendar.get_today` | Get today's events |
| `calendar.get_events` | Get events for N days |
| `calendar.create_event` | Create a new event |
---
## Database Schema
The database is SQLite with WAL mode. Schema is created on startup; new columns are added via idempotent migrations.
```sql
-- Users (from Google OAuth)
users (
id, google_id, email, display_name, profile_picture_url,
public_gallery_token, -- UUID for public gallery share link
llm_provider, -- 'Gemini' or 'ChatGPT' (default: 'Gemini')
openai_api_key_encrypted, -- AES-256-GCM encrypted
gemini_api_key_encrypted, -- AES-256-GCM encrypted
active_prompt_id, -- FK to llm_prompts.id
created_at, updated_at
)
-- Linked devices
devices (id, user_id, device_name, device_model, last_seen_at, last_heartbeat_at, linked_at, created_at)
-- QR code link tokens (single-use, 5 min expiry)
link_tokens (id, user_id, expires_at, used_at, used_by_device_id, created_at)
-- Encrypted Google OAuth tokens
oauth_tokens (id, user_id, provider, access_token_encrypted, refresh_token_encrypted, ...)
-- Scene images from glasses
glass_images (
id, device_model, device_instance_id,
scene_description, -- May be empty; server-side LLM fills it if blank
original_filename, thumbnail_filename,
captured_at, received_at, created_at,
image_width, image_height, -- Pixels (nullable)
user_id, -- FK to users.id (nullable for legacy rows)
llm_provider -- Which LLM generated the description
)
-- User-defined LLM prompts
llm_prompts (id, user_id, name, text, created_at)
-- Third-party MCP servers registered via console
third_party_mcp_services (id, name, description, owner_user_id, endpoint_url, auth_type, auth_config_encrypted, tools_json, ...)
```
---
## Directory Structure
```
src/
├── index.ts # Entry point — starts HTTP + WebSocket servers
├── config/
│ ├── env.ts # Zod environment validation (exits on invalid config)
│ └── mcp-services.ts # Built-in MCP service definitions
├── server/
│ ├── express.ts # Express app — middleware, route mounting, error handling
│ ├── sessions.ts # In-memory session store for console auth
│ └── websocket.ts # WebSocket server with zombie connection cleanup
├── routes/
│ ├── auth.ts # Google OAuth flow
│ ├── console.ts # Web console dashboard
│ ├── devices.ts # Device management API
│ ├── gallery.ts # Image gallery (authenticated + public)
│ ├── health.ts # Health check
│ ├── images.ts # Image upload + retrieval API
│ ├── link.ts # QR code token generation
│ ├── mcp.ts # MCP services API
│ └── settings.ts # LLM settings page
├── services/
│ ├── auth.service.ts # OAuth + token refresh mutex
│ ├── device.service.ts # Device CRUD
│ ├── image.service.ts # Image save/list/delete + thumbnail generation
│ ├── link.service.ts # QR code token handling
│ ├── llm.service.ts # LLM settings CRUD + callLlm() (Gemini / ChatGPT)
│ ├── mcp-proxy.service.ts # MCP tool execution with input coercion
│ └── mcp-registry.service.ts # Third-party MCP server registry
├── websocket/
│ ├── handler.ts # Message routing with progress feedback
│ ├── protocol.ts # Message type definitions
│ └── connection.ts # Connection tracking
├── db/
│ ├── index.ts # SQLite connection + WAL mode
│ └── schema.ts # CREATE TABLE + ALTER TABLE migrations
├── utils/
│ ├── crypto.ts # AES-256-GCM encrypt/decrypt + token generation
│ ├── logger.ts # Pino structured logging
│ └── cache.ts # LRU cache for MCP tool results
└── types/
├── api.ts # REST API types
├── mcp.ts # MCP types
└── websocket.ts # WebSocket message types
```
Data files are written to `./data/`:
- `./data/glasscloud.db` — SQLite database
- `./data/images/` — Original uploaded images
- `./data/thumbnails/` — 400×400 JPEG thumbnails (generated by Sharp)
---
## Security
- **OAuth tokens** encrypted at rest with AES-256-GCM using `ENCRYPTION_KEY`
- **LLM API keys** encrypted at rest with the same key
- **Device JWTs** signed with `JWT_SECRET` (HS256); verified on every API request
- **Link tokens** stored as SHA-256 hash only; single-use; 5-minute expiry
- **Rate limiting** on all `/api/*` routes (configurable via env)
- **Helmet** security headers on all responses
- **Public gallery tokens** are random UUIDs; regenerating one instantly revokes the previous link
---
## License
MIT
