aiko-forge

# Aiko Forge Turn one anime image into a layered, rigged, animatable companion, rendered by a runtime you control. No proprietary `.moc3`, no Cubism license, no GPU on the serving box. The heavy machine-learning steps run on hosted APIs or a rented GPU; the rig itself is deterministic CPU code. The models this factory produces play live in [Aiko](https://github.com/0xAnni/aiko), our open-source companion app. All eleven characters that ship inside Aiko came out of this pipeline, and the eleventh, mira, was minted end to end by somebody else's AI agent over MCP: one sentence in, a living character out, no human touching a file in between.  That loop is the actual shipping renderer at real speed: layered idle sway, gaze, one asymmetric blink, hair riding its spring bones. This repo is the working pipeline behind the build. It is honest about where the output is good and where it is still rough. See [`FINDINGS.md`](FINDINGS.md) for the running log and [`RESEARCH-and-architecture.md`](RESEARCH-and-architecture.md) for the full design and the format decision. ## Why this exists Commissioned Live2D rigging runs about $100 to $2,000. The layer-decomposition half of that work is now a solved research problem (See-through, SIGGRAPH 2026), the rendering half is a commodity, and the missing middle (layers to a real rig) is deterministic code. Aiko Forge stitches those into something that plays in a browser companion for cents. The honest ceiling: the auto output is a 2.5D companion (parallax, blink, lip-sync, breathing, hair sway), not a hand-rigged Cubism model with true multi-angle head turns. It is good enough for a talking half-body companion. It is not a drop-in for a $300 commission, and the pipeline says so. ## The three tiers - **Tier A, instant talking-head.** One image driven by expression parameters via `fal-ai/live-portrait/image`. Ships in hours. Honest limit: on anime it turns the head but barely opens the mouth and will not blink (its landmark model is trained on human faces). Use it as a head-motion effect, not the talking core. - **Tier B, rig-ready layers.** See-through decomposes one image into \~23 occlusion-inpainted layers plus a PSD, on a rented GPU. Sellable to human riggers on its own. - **Tier C, full auto-rig.** Layers to auto-mesh to template deformers to our open `.aiko` bundle, rendered live in Aiko. The defensible core. This is where the mouth actually opens and the eyes actually blink, because we own the geometry. ## What runs where (no GPU on the build server) | Step | Where | |---|---| | Character generation (Seedream v5 pro via fal.ai) | hosted API | | Whole-character matte (`rembg` isnet-anime) | CPU / build server | | Tier A talking-head (`fal-ai/live-portrait/image`) | hosted API | | Tier B decomposition (See-through) | rented GPU (Modal) | | Occluded-gap fill (LaMa / Flux Fill) | hosted API (optional) | | Auto-mesh + template deformers | CPU / build server | | Package `.aiko` + PSD | CPU / build server | | Render + live drive | user's browser | ## Where to put the two repos `rig_cast.py` takes its own folder, steps one level up, and looks for `aiko/static/models`. So the two clones have to share a parent, and the app's folder has to be named exactly `aiko`: ``` whatever-you-like/ ├── aiko-forge/ <- you run rig_cast.py here └── aiko/ <- finished characters land here └── static/models/nova/ ``` Cloning the app *inside* this repo does not count, and neither does renaming it. If you used GitHub's **Download ZIP** button rather than `git clone`, rename both folders before you start. GitHub names a zip after the branch, so you will have `aiko-main` and `aiko-forge-main` sitting next to each other, and nothing will find anything: ``` mv aiko-main aiko mv aiko-forge-main aiko-forge ``` Any other layout works too, you just have to say where: ``` AIKO_MODELS_DIR=/somewhere/aiko/static/models python rig_cast.py nova ``` The script will not create that path for you. If it cannot find a real one it stops and says so, rather than letting `copytree` invent a directory and drop a paid GPU run somewhere nobody is watching. ## Quickstart First time through, the whole setup is nine steps. The first six are one-time account plumbing; only steps 7 and 8 repeat per character: 1. `git clone` this repo and [the companion app](https://github.com/0xAnni/aiko) into the same parent folder (see [Where to put the two repos](#where-to-put-the-two-repos)) 2. create the venv and install `requirements.txt` (below) 3. put your `FAL_KEY` in `.env` 4. `modal token new` 5. create the `huggingface` Modal secret, deploy the worker, download the weights (see [Keys and accounts](#keys-and-accounts)) 6. install the app's own requirements so you have somewhere to watch the result 7. get an image into `samples/output/cast_gen/<id>.png` (generate one, or bring your own) 8. `python rig_cast.py <id>` 9. run the app and pick the character Steps 1 to 6 happen once per machine, step 9 once per session. After that a new character is steps 7 and 8, and they run unattended: no hand-editing of layers, no manual rigging, no cleanup pass. ```sh python -m venv .venv # Windows .venv\Scripts\pip install -r requirements.txt copy .env.example .env # macOS / Linux .venv/bin/pip install -r requirements.txt cp .env.example .env ``` Then put your `FAL_KEY` in `.env`. Python 3.10 or newer. Generate a character with the same generator the cast used (Seedream v5 pro, prompts pinned by [`STYLE.md`](STYLE.md) and [`cast_prompts.md`](cast_prompts.md)), then decompose and rig it: ``` python gen_seedream.py aiko # -> samples/output/cast_gen/aiko.png python orchestrator/tierB_decompose.py samples/output/cast_gen/aiko.png python rigger/build_aiko.py # -> samples/output/rig-v2/ python rigger/render_still.py # -> samples/output/proof/*.png ``` `gen_seedream.py` only knows the ten characters that have prompt blocks in `cast_prompts.md`; making one of your own is the next section. `python rig_cast.py all` runs that whole chain for every one of them and installs each bundle into the companion app. The one source image committed here is `samples/output/cast_gen/mira.png`, because the article uses it. That also makes `python rig_cast.py mira` the cheapest end-to-end test of your setup: it skips generation and goes straight to decomposition and rig, so it tells you whether your Modal worker is really working before you spend anything on a character you care about. ## Making a character that is YOURS The commands above rebuild characters this repo already knows. To mint somebody new, the only thing the pipeline needs is an image at `samples/output/cast_gen/<id>.png`. Where that image comes from is up to you, and there are three honest ways: **Let an agent do it.** Set up the MCP server (see below) and hand it a sentence. It appends the style bible for you, so an agent cannot order a design the rig chokes on, and `create_character` runs generation and rigging both: ``` create_character("nova", "tall, silver bob, red eyes, navy jacket over cream") ``` **Write the prompt yourself.** Add a block for your character to [`cast_prompts.md`](cast_prompts.md) beside the ten already there. Your block is identity only: face, hair, palette, wardrobe. The SHARED TAIL at the top of that file is what gets appended to it, and that tail is the part that keeps a design riggable, so leave it alone. Then: ``` python gen_seedream.py nova && python rig_cast.py nova ``` **Bring your own art.** Skip generation entirely: drop a front-facing portrait at `samples/output/cast_gen/nova.png` and run `python rig_cast.py nova`. Read [`STYLE.md`](STYLE.md) first. A barely-there closed mouth, a low open neckline, hair in separated masses and nothing perched on the head are not aesthetic preferences, they are what survives decomposition. Ids are lowercase, 2 to 24 characters, letters digits `_` `-`, starting with a letter. Whatever you pick becomes the folder name in the app. ## Seeing it alive A bundle is layers and boxes. To watch it move, install the companion app beside this one, because that is where the renderer lives: ``` git clone https://github.com/0xAnni/aiko ../aiko cd ../aiko && pip install -r requirements.txt && python server.py ``` The app ships eleven finished characters, so it is also the fastest way to see what the pipeline produces without spending anything: clone it, paste an Anthropic key, and one of them is talking to you. `rig_cast.py` installs into `../aiko/static/models` by default; see [Where to put the two repos](#where-to-put-the-two-repos) if your checkouts do not sit that way. **Why the app and not something in here.** A `.aiko` bundle deliberately carries no behaviour: layers, boxes, anchors, depths, and nothing about how any of it moves. All the motion lives in the renderer, which is what lets a character minted tomorrow move like the cast shipped today, and lets an improvement reach every model ever built on the next page load. So the bundle needs a renderer, and the real one is three thousand lines in `aiko/static/avatar.js`: spring-bone hair, whole-figure weight drift, a painted mouth interior with teeth, tongue and five vowel shapes, emotions, gestures, and a dozen repairs applied at load. There is a renderer in this repo too, but be clear about what it is for. The older experiments still work and stay for the receipts: `orchestrator/ gen_character.py` is the retired flux generator, and `orchestrator/ tierA_talkinghead.py` plus `rigger/local_split.py` are the Tier A path: ``` python orchestrator/tierA_talkinghead.py # -> samples/output/tierA_talking.mp4 python rigger/local_split.py # -> layers + character.aiko.json ``` ## Proving the format, without the app `runtime/forge-proof/` is a 285-line reference player, and `rigger/render_still.py` is the same thing again in Python with no browser at all. They exist to prove one claim: that the same bundle composites identically in a browser canvas and in a headless CPU compositor. That is the test of whether a format is real or secretly belongs to one runtime. ``` python rigger/render_still.py # no browser needed python -m http.server 8777 --directory . # open http://127.0.0.1:8777/runtime/forge-proof/index.html ``` Drag the sliders, or hit **Play** to watch it talk and blink. The panel drives the same named parameters a companion AI writes each frame. What this player is NOT is the finished look. It has no mouth interior, no visemes, no hair physics worth the name. If you judge the output by this page you will badly undersell what came out of the pipeline. Install the app. ## MCP: let any AI agent mint characters Character creation is exposed as an MCP server, so an agent (Claude Code, a bot, anything speaking MCP) can go from one sentence to a living character without a human touching a file: ``` claude mcp add aiko-forge -- .venv/Scripts/python.exe mcp_server.py # Windows claude mcp add aiko-forge -- .venv/bin/python mcp_server.py # macOS / Linux ``` Tools: `list_characters`, `generate_character(id, description)` (the style bible tail is appended automatically), `rig_character(id)` (See-through on Modal + build + ship, 5-8 min), `create_character` (both in one call), `swap_outfit` / `restore_outfit`. These start where the setup ends. There is deliberately no MCP tool for creating the venv, deploying the Modal worker or pulling the weights, because those are account-level actions that need your browser and your keys. Do the six one-time steps in [Quickstart](#quickstart) yourself, and from then on an agent can mint characters all day without you. ## Outfit swap (rig-safe re-dressing) ``` python outfit_swap.py aiko "a deep crimson jacket with gold piping" python outfit_swap.py aiko --restore ``` Seedream repaints the topwear layer's colors and materials while the layer's original alpha is kept, so the silhouette stays exactly as it was, and with it the whole rig. Wardrobes for the cast at almost zero cost. ## Keys and accounts Only one key goes in `.env`: - `FAL_KEY` (required). Character generation, Tier A, outfit swap. Tier B, the See-through decomposition on a rented GPU, does NOT read a token from `.env`. It calls a worker you deploy to your own Modal account, and that worker reads Hugging Face credentials from a Modal Secret rather than from your machine. This is the part that catches people: `rig_cast.py` looks up a **deployed** function by name, so until you have run the deploy below it fails with a not-found error at the most expensive step of the chain. Four one-time commands: ``` modal token new # opens a browser, writes ~/.modal.toml modal secret create huggingface HF_TOKEN=hf_... # the worker reads the token from here modal deploy gpu-worker/modal_app.py # publishes the "aiko-forge" app modal run gpu-worker/modal_app.py::download_weights # ~13.4 GB into a Modal Volume, once ``` The secret has to be named exactly `huggingface`; that is the name the worker asks for. The weights download is slow but happens once per account: it fills a Modal Volume that every later run mounts instead of re-fetching. After these four, `rig_cast.py` runs unattended. `AIKO_MODELS_DIR` (optional) sets where finished bundles are installed, for layouts other than the default one described in [Where to put the two repos](#where-to-put-the-two-repos). ## Status The full chain is proven end to end on eleven characters. A generated image (\~$0.03) goes through See-through on a Modal L4 (\~$0.08, 18 semantic parts with depth ordering and per-side eye components), through the deterministic CPU rig builder, into an open `.aiko` v2 bundle that lives in the browser renderer. Marginal cost per character: roughly a dime. The eleventh was ordered by somebody else's AI agent over the MCP server, from one sentence, with no human touching a file afterwards. That is the test that matters for a factory: it works for a stranger, not just for its author. The See-through face layer arrives fully inpainted behind the hair, eyes and mouth, with subtle closed-lid hints where the eyes were, so a blink is an honest vertical squash of the real eye layers and reads like a hand-rigged model. The bootstrap decomposer (`rigger/local_split.py`) remains as a zero-account fallback and the renderer supports both bundle versions. ## Where it still loses to a human rigger Measured, not guessed, and none of it is hidden: - **No true head turn.** 2.5D parallax from a single front image, so there is no profile. Generating extra views and swapping between them is possible and was costed; it is not built. A commissioned model wins this outright. - **A regular mesh, not one drawn to the art.** The face bends on an 8x12 grid and the hair on per-strip chains, against a human rigger's adaptive mesh. Worth knowing: an automatic per-strand mesh was investigated and rejected on evidence, because 99.8% of a hair layer's alpha is one connected component, so "each strand" is not recoverable from the data at all. - **One motion language for the whole cast.** Every character inherits the same physics; a human tunes each one by hand. There is now an optional per-character gain table in the renderer, deliberately empty until it earns its place. - **Documented failure modes.** Tied twintails, high ponytails, heavy frills and chibi proportions break decomposition, and the style bible routes around them rather than pretending otherwise. The failure gallery is next. What is NOT on that list, because it shipped: per-vertex mesh warp, iris gaze and wander, brow raise and furrow, a fully painted talking mouth with cavity and teeth, spring-bone hair on roughly a hundred bones, whole-figure weight drift, audible laughter, webcam face tracking, and a three-character stage that hosts a live show. ## Licence MIT, in [`LICENSE`](LICENSE). Take it, fork it, sell what you make with it. Everything in this repository is mine and is MIT. One carve-out, and it is not about the code. The cast this factory built (the eleven characters in the companion app, including mira, whose source image ships here) is the cast of my own show, licensed for personal use in the app's [CHARACTERS-LICENSE.md](https://github.com/0xAnni/aiko/blob/main/CHARACTERS-LICENSE.md). Run them at home, but stream with a cast you forge yourself. What you forge is yours entirely, and this repo exists so that forging one costs eleven cents. Nothing here redistributes anybody else's code. The decomposer is [See-through](https://github.com/shitagaki-lab/see-through), Apache-2.0, and it belongs to its authors: the Modal worker clones it at image-build time and pins a commit, so their code never lands in this repo. The companion app takes the same line with Live2D, loading the Cubism core from Live2D's own CDN and shipping none of it, because that core is under Live2D's licence and not a thing anyone may put in an MIT repository.