openfield

<div align="center"> # OPENFIELD **A self-hosted, cinematic AI generation studio — an open Higgsfield.** Image, video, 3D, and audio — plus cutout, upscale, reframe, edit, and more — from one prompt-first studio. Driven by humans through a web UI and by coding agents through MCP. Runs on a bundled **mock** engine locally and a real **SwarmUI** engine on a GPU box — you flip between them with one env var. <em>MIT licensed · Next.js + FastMCP · bring your own GPU</em>  </div> --- ## What it is - **A full generation stack** — **create** with Image, Video, **3D** (image/text → GLB), Audio (voice / music / SFX), Consistent-subject, and Product-placement; **transform** with Cutout, Upscale, Reframe, Edit (mask inpaint), Lip sync, Motion transfer, and Video upscale. One prompt-first composer, driven by a data spec — not a wall of one-off screens. Most models are **opt-in**, each carrying an **untested** badge until verified on real hardware. - **A studio, not a form** — every render is kept in a gallery you can chain from: reuse a prompt, **animate a still into video**, remove a background, upscale, reframe, score it with audio, or download. - **Presets** — a card wall of authored cinematic looks (camera moves, lighting, portrait, product, VFX, seamless-loop backgrounds) with a fill-in `{subject}` slot. - **Agent parity via MCP** — anything the UI can do, a coding agent can do through a FastMCP (streamable-HTTP) server that is a thin client of the same API. - **A model registry, not hardcoded models** — add a model by adding one JSON entry. - **Correct by construction** — per-model *required injections* live in the registry, so a video can never render as a blank teal clip (the LTX-2.3 failure mode) from any surface — UI or agent. | | | |---|---| |  |  | | **Presets** — click a card, fill the subject, shoot | **Gallery** — every render, with prompt/seed/settings | --- ## Where OPENFIELD goes beyond Higgsfield An open studio, not a hosted catalog. Where it exceeds the reference product: - **True self-hosting.** Your weights, your box, your data. No per-generation fees, no credits, no rate limits, no vendor holding the render queue. Run it on a rented GPU or your own tower — the studio is the same. - **3D generation.** Image → GLB and text → GLB (TRELLIS.2, MIT-licensed) sit right next to image and video in the same composer and gallery. The reference product ships **no** 3D at all. - **Agent-native by design (MCP).** The *whole* studio is scriptable by coding agents over a FastMCP server — full parity with the UI, zero drift, because the MCP server is a thin client of the same API. A `motion-spec` MCP tool even hands an agent a structured motion brief to build in GSAP / SVG / Lottie. - **An open preset + model registry.** Models and looks are **data** you can PR — one JSON entry adds a model, one JSON file adds a cinematic preset. It's a community catalog you extend, not a closed one you're stuck inside. --- ## Quickstart — local, on the mock engine (3 commands, zero config) ```bash cd ui npm install npm run dev ``` Open **http://localhost:3000**. No API keys, no `.env`, no GPU. The mock engine serves bundled cinematic sample media with real queue behaviour, so every surface and every flow (generate → gallery → **animate** → cutout) works exactly as it will against real hardware. Requires Node 20+. ### Give an agent the same powers (optional) ```bash cd mcp python3.11 -m venv .venv .venv/bin/pip install -r requirements.txt OPENFIELD_API_URL=http://localhost:3000 .venv/bin/python server.py ``` The MCP server comes up at **http://127.0.0.1:8000/mcp** (streamable-HTTP) — add it to Claude Design → Settings → Connectors. Tools: `generate_image`, `generate_video`, `remove_background`, `use_preset`, `list_presets`, `list_models`, `get_job`, `list_active_jobs`. List them without a client: `.venv/bin/python server.py --print-tools`. --- ## How it works **One driver.** The Next.js app (`ui/`) *is* the backend: it owns the SwarmUI driver (mock + real), the model registry, inference resolution, the SQLite gallery, and the async queue, all behind a REST API. The Python MCP server (`mcp/`) makes no engine decisions — it forwards to that API. So there is exactly one place where model rules, persistence, and the video-safety guarantee live, and no drift between what a human and an agent can do. **Mock vs real is one env var.** `SWARM_URL` unset → the bundled mock engine; `SWARM_URL=http://127.0.0.1:7801` → a real SwarmUI. Same driver, queue, gallery, presets, and registry either way. **The model registry** (`models/registry.json`) is the single source of truth. Each entry declares a model's overridable `defaults` and its `required` injections — forced params and LoRA tags that win over everything. The resolver is generic and reads the registry, so: ```jsonc // the LTX-2.3 teal-fix is just data — no special-casing in code: "required": { "params": { "steps": 8, "cfgscale": 1 }, "loras": [{ "name": "ltx-2.3-22b-distilled-lora-384-1.1", "strength": 0.8 }] } ``` A model that isn't in the registry still works, on safe generic defaults — you're never blocked. Presets never hardcode a model, so they run on whatever stack you have installed. ### Add a model — from the UI or one JSON entry The **Models** page (a Model Library) is the easiest way: see every model with live status (**ready / downloading / available**), **install or download with one click**, **browse HuggingFace by type** to discover new weights, and **add your own** (a persisted overlay in `models/user-registry.json` that survives updates). Under the hood it's still the registry — you can also edit `models/registry.json` directly (`install: true` to have `server/setup.sh` download it). Full field-by-field guide: **[docs/adding-models.md](docs/adding-models.md)**. The shipped stack: | Model | Capabilities | Notes | |-------|------|-------| | `flux2-dev-fp8mixedfromscaled` | image | FLUX.2 — cfg 1 enforced; **verified on hardware** | | `ltx-2.3-22b-dev` | video | LTX-2.3 — distilled LoRA + steps 8 enforced; **verified on hardware** | | `supir` | upscale | SUPIR diffusion upscaler (ComfyUI workflow) | The first two are verified end-to-end on a real GPU; the rest of the stack is **opt-in**, each carrying an *untested* badge until verified on real hardware: TRELLIS.2 (3D), Chatterbox / Stable Audio / MMAudio (voice / music / SFX), FLUX.2 Fill (reframe + edit), PuLID (consistency), FLUX.2 multi-reference (placement), InfiniteTalk / LatentSync (lip sync), Wan 2.2 Animate / MimicMotion (motion transfer), SeedVR2 (video upscale), plus best-effort SDXL and Wan 2.2 T2V. Flip `install: true` on any entry to add it to the download stack. --- ## Deploy to a GPU (the real engine) From your Mac, once you have SSH to a fresh box: ```bash ./server/deploy.sh 'ssh -p <PORT> root@<host>' ``` `deploy.sh` rsyncs the repo and runs `server/setup.sh`, which **detects its environment** — DATA_DIR (auto-prefers big mounted volumes like AutoDL's `/root/autodl-tmp` or RunPod's `/workspace`), supervision (systemd units / Docker Compose / tmux), and downloads exactly the models the registry marks `install:true`. **Fully hands-off — no browser wizard.** setup.sh drives SwarmUI's own installer headlessly (`server/swarm-install.py`): one ComfyUI backend, bound to localhost. It then brings up the Studio + MCP and tunnels **only those two** — SwarmUI and ComfyUI stay private (never exposed). Model downloads (~72 GB) plus a fast aria2 **encoder pre-fetch** overlap the rest, so the first generation of each model isn't stuck on SwarmUI's slow single-stream encoder download. When it finishes it prints the Studio + MCP URLs and you're done — generation works as soon as a model + its encoder land. Update everything later with `server/update.sh`. Guides: **[deploy overview](docs/deploy/)** — **[AutoDL](docs/deploy/autodl.md)** · **[RunPod](docs/deploy/runpod.md)** · **[generic Linux / Docker](docs/deploy/generic-linux.md)**. There's also a `docker-compose.yml` (UI + MCP; point `SWARM_URL` at your SwarmUI). --- ## Taste OPENFIELD is meant to look like a film studio, not a SaaS admin: warm near-black chrome, a single amber accent, self-hosted **Fraunces / Hanken Grotesk / JetBrains Mono**, a custom icon set, film grain, motion with a job. The generated media is the only real color on screen. Design tokens live in `ui/src/app/globals.css` — derive components from them and keep the discipline (no purple gradients, glassmorphism, emoji icons, or generic rounded-2xl cards). --- ## Project layout ``` openfield/ ├── ui/ Next.js studio — driver, registry resolver, gallery, queue, API ├── mcp/ FastMCP (streamable-HTTP) server — thin client of the UI API ├── models/ registry.json — the single source of truth for models ├── presets/ authored looks (plain JSON, {subject} slots) ├── server/ setup.sh (env-detecting bootstrap) · deploy.sh (from your Mac) │ swarm-install.py (headless SwarmUI install) · update.sh ├── docs/ deploy/ guides · adding-models.md · media/ screenshots ├── Dockerfile · mcp/Dockerfile · docker-compose.yml ├── DECISIONS.md every product/engineering call and why └── .env.example every environment variable, documented ``` --- ## Contributing **Presets and the model registry are where community PRs are most welcome** — a new cinematic preset is a JSON file; a new model is a registry entry. See **[CONTRIBUTING.md](CONTRIBUTING.md)** and **[docs/adding-models.md](docs/adding-models.md)**. The golden rule: model-specific behaviour goes in `models/registry.json`, never in code — the resolver stays generic. ## License MIT — see [LICENSE](LICENSE).