build-small-hackathon/Hackathon-IA-VisualNovel

1---2title: "How We Built Ephemeral Hearts - a Live-Improvised Anime Visual Novel with Several Small AIs"3thumbnail: /blog/assets/ephemeral-hearts/thumbnail.png4authors:5  - William Hubaux ; Github, [WillHCode](https://github.com/WillHCode)6  - Lorenzo Lepoivre ; Github, [SupPepper](https://github.com/LorenzoLepoivre)7date: "2026-06-09"8tags:9- hackathon10- visual-novel11- multi-agent12- diffusion13- gradio14- modal15- lora16---17 18# How We Built Ephemeral Hearts - a Live-Improvised Anime VN with Several Small AIs19 20*Build Small Hackathon - June 2026*21 22---23 24## Table of contents25 261. [Who we are and why we built this](#who-we-are)272. [The multi-agent setup: three roles, three model families](#multi-agent)283. [What we discovered along the way](#discoveries)294. [The "model proposes, code disposes" pattern](#pattern)305. [Features that emerged naturally](#features)316. [The journey from local to Modal](#local-to-modal)327. [One codebase, three runtimes: local, Modal, Space](#three-runtimes)338. [llama.cpp vs transformers vs ZeroGPU: what actually changes](#backends)349. [Architecture, in depth](#architecture)3510. [Field notes: the problems we hit and how we fixed them](#field-notes)3611. [Stack](#stack)3712. [What this could become](#vision)38 39---40 41## Who we are and why we built this <a name="who-we-are"></a>42 43We are two developers, and we both already work on a production AI project with RAG pipeline, local models via HuggingFace Transformers, multi-agent LLM system - all aimed at real business use cases. So we were not complete beginners when it came to language models. But this hackathon was something different: how do you put all of this together in a **fun** way, with creative constraints, and explore territories we had not seriously touched yet - image generation, voice models, fine-tuning diffusion models with LoRA, and Modal for on-demand GPU deployment.44 45The base idea: an anime visual novel where *everything* is generated live. No pre-written script, no fixed assets. Every character, every backdrop, every line of dialogue - conjured on the fly by small models. The hackathon imposed ≤ 32B total parameters and a Gradio app on a HF Space. Perfect. We figured that if the constraints forced us to be creative about the architecture, we might as well build one we would be proud to explain.46 47The result: **Ephemeral Hearts** - an anime dating VN where the story, dialogue, characters, and art are all generated in real time. Nothing is pre-scripted.48 49---50 51## The multi-agent setup: three roles, three model families <a name="multi-agent"></a>52 53The first real design question was: how many models, which ones, and how do you make them collaborate?54 55We defined three distinct **roles**, each handled by a different model family:56 57> **🧵 The Weaver** - the game director. It reads the current scene, the character sheets, the recent conversation, and the player's input. It decides what changes structurally this turn: does the scene shift? Does a new character arrive? What is the relationship delta? It outputs a structured JSON block of *directives*.58 59> **🎭 The Voices** - the actor. It speaks *as* the active character - their speech quirks, their current mood, their secrets. One to three sentences, anime style, never breaking the fourth wall.60 61> **🎨 The Painter** - it generates backdrops and character sprites. Default backend: SDXL-base-1.0 distilled down to 4 inference steps with the ByteDance SDXL-Lightning LoRA (an SDXL-Turbo backend exists as an alternative, and `VN_IMAGE_LORA` accepts a custom style LoRA on top).62 63> **👂 The Ear** - voice transcription. Whisper (large-v3-turbo by default), CPU via faster-whisper, good enough for short clips.64 65> **🔊 The Larynx** - speech synthesis. Kokoro-82M via ONNX gives every character a voice, picked once at creation from their written voice description and frozen forever (same character = same voice).66 67The key subtlety: **The Weaver and The Voices share a single LLM** (Qwen3-14B). Same weights, two system prompts, one grammar-constrained call per turn. They write the stage directions and the dialogue lines simultaneously. A second LLM call per turn would have doubled perceived latency - and the two roles are so tightly coupled that sharing context is actually an advantage.68 69### Architecture diagram70 71```72 Player input (text or voice)73         |74         v75 +-------------------+76 |   THE EAR  👂     |  Whisper-small77 |  voice -> text    |  (CPU - faster-whisper)78 +--------+----------+79          |80          v81 memory.assemble_context()82 +---------------------------------------------+83 |  summary + current scene + character sheets |84 |  + last k exchanges  (fixed token budget)   |85 +------------------+--------------------------+86                    |87                    v88 +------------------------------------------------------+89 |          ONE grammar-constrained call                |90 |                                                      |91 |   THE WEAVER  🧵  (director)                         |92 |   + THE VOICES 🎭  (actor)                           |93 |                                                      |94 |   Qwen3-14B                                          |95 |   +-- local : llama.cpp  (Metal / ROCm)              |96 |   +-- cloud : Modal A10G  (VN_LLM_BACKEND=modal)     |97 +----------------------+-------------------------------+98                        |99                        v100                  DirectorOutput101        +---------------------------------+102        |  dialogue  .  emotion          |103        |  scene_change                  |104        |  relationship_delta            |105        |  new_character / exit_char     |106        |  set_music  .  ending          |107        |  npc_relation_deltas           |108        +--------------+-----------------+109                       |110                       v111          state.apply_directives()112          <- THE ONLY MUTATOR113          GameState updated114          .md dream-memory written115                       |116          +------------+-------------+117          v                          v118 +-----------------+        +-----------------+119 |  THE PAINTER 🎨 |        |    ViewState     |120 |                 |        |  -> frontend     |121 |  SDXL-Turbo     |        |  (Gradio Server) |122 |  + anime LoRA   |        +-----------------+123 |                 |124 |  +- local :     |125 |  |  diffusers   |126 |  |  (Metal/ROCm)|127 |  +- cloud :     |128 |     Modal A10G  |129 +--------+--------+130          |131          v132   backdrop.png  (cached by seed)133   sprite.png    (cached by character + mood)134```135 136**Total: ≈18B parameters. Well under the 32B budget.**137 138| Role | Model | Params |139|---|---|---|140| The Weaver + The Voices (shared) | Qwen3-14B | ~14B |141| The Painter | SDXL-base-1.0 + Lightning LoRA | ~3.5B |142| The Ear | Whisper large-v3-turbo | ~0.8B |143| The Larynx | Kokoro-82M (TTS) | ~0.08B |144| **Total** | | **≈18B** |145 146---147 148## What we discovered along the way <a name="discoveries"></a>149 150We already had experience with LLMs and HuggingFace. But this project pushed us into three territories we had not seriously explored before.151 152### Image models - SDXL-Turbo and LoRA153 154This is clearly what surprised us the most. We knew image generation existed, we had played with online demos, but actually integrating it into a real pipeline - managing prompts, seeds, caching, negative prompts, and visual consistency - is a genuine craft in itself.155 156Choosing a few-step distilled model over a full diffusion pipeline was deliberate: 1 to 4 inference steps instead of 50. Slightly lower quality, but in a VN where the image arrives *while* the player is reading the dialogue, "fast and impressionistic" beats "slow and polished" every single time. We started with SDXL-Turbo and eventually settled on SDXL-base-1.0 + the ByteDance Lightning LoRA as the default - same philosophy, noticeably better quality at 4 steps (both backends remain in the code).157 158What we had never done before: **LoRA fine-tuning**. We trained an anime LoRA on SDXL-Turbo to lock in the visual style - color palette, character rendering, line quality. The result: even as prompts vary wildly from scene to scene, the style stays consistent. That is exactly what a VN needs. The LoRA is published on the Hub (`VN_IMAGE_LORA`) and loads automatically at startup.159 160Another revelation: **seed pinning**. Every character sprite is generated **once** per mood, with a fixed seed, and cached forever. Same character, same scene, same expression = same image, always. The visual consistency this gives does more for immersion than a higher-quality model would.161 162### Voice models - Whisper163 164Whisper-small for voice input was the easiest onboarding. CTranslate2 via faster-whisper, CPU, works on both our machines without a dedicated GPU. The only gotcha: CTranslate2 has no ROCm backend, so on our AMD box it runs on CPU regardless. For 5-10 second clips it is more than sufficient.165 166### Modal - GPU on demand167 168This was the real infrastructure discovery of this hackathon.169 170We were used to deploying on HF Spaces or fixed GPU servers. Modal is a completely different mental model: you write a Python function, decorate it with `@app.function(gpu="A10G")`, and Modal handles everything else - containerization, scaling, cold starts, per-second billing. For a hackathon where you have no idea what load you will get, it is perfect.171 172The Gradio Space stays lightweight (CPU only), model calls go out to Modal A10G containers. The `VN_LLM_BACKEND=modal` environment variable switches the backend without touching any game logic. Same interface, different implementations.173 174---175 176## The "model proposes, code disposes" pattern <a name="pattern"></a>177 178The naive approach: let the LLM generate free prose and parse the result. This breaks immediately with small models - they drift, contradict themselves, forget who is on stage.179 180Our approach: every turn is one grammar-constrained call that returns a typed Pydantic `DirectorOutput`:181 182```python183# Schema is derived directly from the Pydantic model - can never drift184out = llm.complete_json(schema=DirectorOutput.model_json_schema(), prompt=context)185 186# The only place in the entire codebase that touches GameState187effects = state.apply_directives(game_state, out)188```189 190`apply_directives` clamps values, validates references, and silently ignores impossible requests (like removing a character who is not on stage). The LLM never touches state directly - it proposes, the code decides.191 192Direct consequence: the entire game loop was testable from day one with `VN_MOCK=1` - deterministic fake LLM, fake painter, zero real models. We built the NPC bond graph, the music system, the ending screens, and the relationship milestones entirely in mock mode before a single real model call ever ran.193 194---195 196## Features that emerged naturally <a name="features"></a>197 198Once the directive pattern was solid, features fell out almost on their own.199 200**NPC-to-NPC relationship graph.** The same `npc_relation_deltas` directive that tracks player-NPC affection also tracks relationships *between* NPCs. Jealousy, rivalry, camaraderie - all of this emerges naturally from the model's tendency to give characters an inner life. We expose it as a directed graph in the Relations tab, with one-word labels ("jealousy", "rivalry", "fond of") coming directly from the model.201 202**Dynamic music system.** Six tracks (calm, romantic, dramatic, mystery, sad, joyful), crossfade on change, all controlled by a `set_music` directive. The model knows the available tracks and only switches for genuine tonal ruptures - not every mood shift. In practice it is surprisingly restrained.203 204**Generated ending screens.** When a relationship hits +100 (romantic confession) or -100 (a falling-out that empties the stage), the model emits an `ending` directive with a poetic 2-4 sentence epilogue. The Painter generates a dedicated illustration - cherry blossoms at golden hour for the warm ending, a rain-soaked empty park bench at night for defeat. No pre-authored cutscenes. The ending art is improvised just like everything else.205 206**Characters that remember you.** A `remember_fact` directive lets the speaker store one short note when the player reveals something personal ("the player plays violin", "the player is named Lorenzo"). Facts are capped at 8 per character and re-injected into their sheet every turn they are on stage - so a character you met ten scenes ago still knows your name. The same progression system unlocks personality traits at affection thresholds (20/40/60), reveals the secret goal at 80, and stages a one-shot "growing close" moment at 50 - all deterministic, all driven by the relationship value the model proposes.207 208**Quality-of-life.** The player can set their name in the setup form (characters address them by it), and a session persisted server-side after every turn powers a "Continue last story" button - including a rebuilt journal with a condensed recap of everything that got compacted away.209 210---211 212## The journey from local to Modal <a name="local-to-modal"></a>213 214We started on our two machines: an Apple M3 Max (Metal backend) and an AMD RX 7900 XTX (ROCm). First surprising fact: ROCm reports itself as `"cuda"` to PyTorch - so our device detection works on both machines without any special-casing. Second surprise: CTranslate2 has no ROCm backend, so Whisper runs on CPU on the AMD machine regardless.215 216For the HF Space we first needed GPU inference without going through ZeroGPU (llama.cpp + ZeroGPU is notoriously unreliable). The solution: Modal for heavy model calls, Gradio for the UI. The Space stays CPU, GPU compute goes to on-demand Modal containers. We later added a third path - running the models directly on ZeroGPU with `transformers` - which turned out to have its own fascinating constraints (see the next two sections).217 218One unexpected practical problem: HF Spaces now uses Xet storage and refuses binary files pushed through git. We had to soft-reset the commit that included the music MP3 files, add them to `.gitignore`, push the clean code, then upload the audio files via `HfApi().upload_file()`. Lesson learned the hard way: code goes through git, binary assets go through the Hub API.219 220---221 222## One codebase, three runtimes: local, Modal, Space <a name="three-runtimes"></a>223 224The same `visualnovel/` package runs in three very different places. Nothing in the game logic knows which one it is in - the backends are selected by `config.py` from environment variables, and every backend implements the same two-method interface (`complete` / `complete_json` for the LLM, `_render` for the Painter).225 226| | **local** | **modal** | **space (ZeroGPU)** |227|---|---|---|---|228| Where the app runs | your machine | your machine (UI) + Modal (GPU) | HF Space |229| LLM engine | llama.cpp, GGUF Q4_K_M, Metal/ROCm | llama.cpp, GGUF Q8_0, A10G container | transformers, full bf16 weights |230| Painter | diffusers in-process | A10G container (RPC) | diffusers in-process |231| Selected by | default | `VN_LLM_BACKEND=modal` | `SPACE_ID` env var (auto-detected) |232| Game state | in-process | in-process | `/tmp` JSON file (workers are stateless) |233| Cold start cost | model load at launch | ~15-30 s container spin-up, then kept warm 10 min | per-worker lazy load inside the first GPU call |234 235A few design decisions make this work:236 237- **Auto-detection over configuration.** HF injects `SPACE_ID` into every Space, so `config.py` flips the LLM backend to `transformers` automatically when deployed - a fresh clone needs zero setup in any environment. Setting `VN_LLM_BACKEND=modal` also chains the Painter to Modal by default, because if you don't have a local GPU for the LLM you don't have one for SDXL either.238- **Modal calls are plain RPC.** `ModalLLM` is a ~20-line proxy: it looks up the deployed class with `modal.Cls.from_name(...)` and calls `.remote()`. The JSON schema travels with every call, which has a lovely property: changing prompts, schemas, or game logic never requires a redeploy - only changes to the container code itself do.239- **Keep-warm matters more than raw speed.** Reloading a 15 GB GGUF mid-conversation is a worse experience than any per-token slowness, so the Modal LLM container keeps a 10-minute `scaledown_window` and the app fires a fire-and-forget warmup ping at server start.240 241---242 243## llama.cpp vs transformers vs ZeroGPU: what actually changes <a name="backends"></a>244 245These three names live at different levels - two are inference engines, one is a runtime - and each one reshaped a different part of the code.246 247### llama.cpp: the grammar is the contract248 249The killer feature for this project: `llama-cpp-python` compiles a JSON schema into a **GBNF grammar** that constrains decoding token by token. The model *physically cannot* emit malformed JSON, invent keys, or put a string where an integer belongs. Our whole "model proposes, code disposes" pattern leans on this.250 251It also exposes full sampling control (`temperature`, `top_p`, `presence_penalty` - the latter became our anti-repetition weapon). Two sharp edges, though:252 253- The grammar cannot *finish* a document when `max_tokens` runs out mid-string. You get perfectly-shaped-but-truncated JSON, and `json.loads` explodes. We wrote a small `close_truncated_json()` repair (close the open string, drop the dangling comma, balance the brackets) so a runaway generation costs a few default-valued fields instead of a crashed turn.254- Qwen3's thinking mode is baked into the GGUF chat template and there is no API switch - the only control is the `/no_think` soft switch appended to the user message.255 256### transformers: no grammar, so trust but verify257 258On the Space there is no grammar constraint. Instead we derive a **JSON skeleton from the same Pydantic schema** and inject it into the system prompt ("Respond with ONLY a valid JSON object. Required structure: ..."), then parse with up to 3 retries and a regex extraction that tolerates markdown fences and stray prose.259 260Two gotchas that cost us real debugging time:261 262- `model.generate()` **silently ignores** `temperature` and `top_p` unless you also pass `do_sample=True`. For weeks the Space was running greedy decoding while we thought we were sampling at 0.7 - and the 3 retries were deterministic, so they retried into the exact same failure.263- Qwen3 sometimes puts its entire answer *inside* the `<think>` block and emits nothing after it. `apply_chat_template(..., enable_thinking=False)` suppresses thinking for structured output, with a fallback that searches inside the think block if nothing comes after it.264 265### ZeroGPU: a runtime, not a backend266 267ZeroGPU is the part that changes your *architecture* rather than your inference code. Each `@spaces.GPU` call can be dispatched to a **different worker subprocess** - in-memory state simply does not survive between two HTTP calls. Three consequences:268 2691. **State lives on disk.** After every mutation, `GameState` is serialized to `/tmp/vn_game_state.json` with an atomic write-then-rename. Every endpoint starts by rehydrating from that file if its own memory is empty. (Bonus: this is exactly the mechanism that later powered the "Continue last story" button.)2702. **Models load lazily.** Loading a 14B model at import time would happen on a CPU-only web worker; instead every backend loads inside the first GPU-decorated call.2713. **Turns are split in two phases** (`/turn_text` then `/turn_images`) so the dialogue can be displayed while the slower image phase runs - and the pending `DirectorOutput` rides along in the state file, because phase 2 might run on a different worker than phase 1.272 273---274 275## Architecture, in depth <a name="architecture"></a>276 277The package is a strict one-way dependency graph - schemas at the bottom, the engine façade at the top, `app.py` as a thin HTTP shell. The golden rule sits in the middle: the LLM returns a typed `DirectorOutput`, and `state.apply_directives` is the **only** function in the codebase allowed to mutate `GameState`.278 279```mermaid280flowchart TD281    UI["frontend/index.html<br/>(custom VN UI, Gradio JS client)"]282    APP["app.py - gradio.Server<br/>thin endpoints: /start_text /start_images<br/>/turn_text /turn_images /transcribe /resume"]283    ENG["engine.py<br/>the façade - owns the session"]284 285    subgraph CORE["visualnovel/ - testable without a server"]286        ORC["orchestrator.py<br/>the Weaver: init / direct_turn / compact"]287        MEM["memory.py<br/>context budget + trimming"]288        CH["characters.py<br/>present-character sheets"]289        PRM["prompts.py<br/>all prompts + JSON schemas"]290        LLM["llm.py<br/>Mock / LlamaCpp / Transformers / Modal"]291        ST["state.py<br/>apply_directives - THE ONLY MUTATOR"]292        PAINT["painter.py<br/>prompt compose + cache + render"]293        STT["stt.py - Whisper"]294        TTS["tts.py - Kokoro"]295    end296 297    UI -->|HTTP| APP --> ENG298    ENG --> ORC299    ORC --> MEM --> CH300    ORC --> PRM301    ORC --> LLM302    ENG --> ST303    ENG --> PAINT304    ENG --> STT305    ENG --> TTS306```307 308A full turn, with the two-phase split that keeps the dialogue snappy:309 310```mermaid311sequenceDiagram312    autonumber313    participant P as Player (browser)314    participant A as app.py315    participant E as Engine316    participant W as Weaver (one LLM call)317    participant S as apply_directives318    participant PA as Painter319    participant T as Kokoro TTS320 321    P->>A: POST /turn_text ("Hello, you are so pretty today")322    A->>E: play_turn_text()323    Note over E: optional: Whisper if voice input324    E->>W: complete_json(context, directive schema)325    W-->>E: DirectorOutput (dialogue, emotion, directives)326    E->>S: apply_directives - clamps, validates, mutates GameState327    E->>E: save .md views + /tmp state (ZeroGPU workers)328    E-->>P: text-only ViewState - dialogue appears NOW329 330    P->>A: POST /turn_images331    A->>E: play_turn_images()332    E->>PA: backdrop(scene) + sprite(char, mood)333    Note over PA: cache hit by (kind, prompt, seed)?<br/>then 0 ms, else 4-step SDXL render334    E->>T: synthesize(dialogue, frozen voice)335    E-->>P: full ViewState - backdrop, sprites, audio336```337 338And the deployment picture - one engine, three homes:339 340```mermaid341flowchart LR342    APP["app.py + visualnovel/<br/>(identical code everywhere)"]343 344    subgraph L["LOCAL - Mac M3 Max / AMD RX 7900 XTX"]345        L1["llama.cpp - Qwen3-14B Q4_K_M<br/>Metal / ROCm, grammar-constrained"]346        L2["diffusers - SDXL + Lightning LoRA"]347    end348 349    subgraph M["MODAL - on-demand A10G"]350        M1["ModalLLMBackend<br/>llama.cpp - Q8_0, kept warm 10 min"]351        M2["ModalPainterBackend<br/>SDXL + Lightning, rembg"]352    end353 354    subgraph Z["HF SPACE - ZeroGPU"]355        Z1["TransformersLLM - Qwen3-14B bf16<br/>JSON skeleton + retries"]356        Z2["diffusers in-process"]357        Z3[("/tmp state file<br/>(stateless workers)")]358    end359 360    APP -->|"default"| L361    APP -->|"VN_LLM_BACKEND=modal<br/>.remote() RPC"| M362    APP -->|"SPACE_ID detected<br/>@spaces.GPU"| Z363```364 365Two invariants hold everywhere:366 367- **One LLM call per turn** (a guarded retry is allowed for repetition or a missed mandatory directive - never a loop).368- **Every Painter render is cached** by `(kind, prompt, seed)`. A character's sprite is generated once per mood and reused forever; seeds are pinned per entity so the same character always has the same face.369 370---371 372## Field notes: the problems we hit and how we fixed them <a name="field-notes"></a>373 374This is the section we wish we had read before starting. Almost every bug below was invisible in the code and obvious in a **saved game file** - our best debugging tool turned out to be exporting the save JSON after each playtest and actually reading it.375 376### 1. Anything optional in the grammar is something the model will skip377 378The single biggest lesson of the project. Our `DirectorOutput` schema had `emotion`, `relationship_delta`, and `new_character` as optional fields with defaults - and the grammar therefore allowed the model to omit them. So it did. Systematically. The symptoms looked like three unrelated gameplay bugs:379 380- every character stayed `"neutral"` forever (so the mood-keyed sprite system never showed a second expression),381- affection stayed at 0 no matter how hard the player flirted (or insulted),382- the "look around" action politely refused to introduce anyone.383 384We tried prompt rules ("you MUST emit new_character"), then an explicit retry quoting the instruction. The model ignored both often enough to ruin the demo. The fix that actually worked: **promote the fields to `required` in the JSON schema** handed to the grammar. For "look around" we went further and built a dedicated schema variant where `new_character` is required *and non-nullable* - introduction guaranteed by construction. Prompt engineering is a suggestion; grammar is a constraint.385 386### 2. Instructions inside quoted player speech get read as... player speech387 388Our action hints ("*the wanderer looks around...*") were concatenated with the player's text and injected as `THE WANDERER NOW SAYS: "<hint> <text>"`. Inside the quotes, the model treated our stage directions as something the player said out loud - and ignored them. Moving the hints to a separate `DIRECTOR NOTE (mandatory):` block outside the quotes changed compliance dramatically. Placement in the context is as important as wording.389 390### 3. Qwen3's thinking mode ate our memory summarizer391 392The rolling summary ("THE TALE SO FAR") is rewritten every ~12 turns by a small free-text LLM call with a 320-token budget. On llama.cpp, Qwen3's chat template enables thinking by default - and the model happily spent the *entire* budget inside an unclosed `<think>` block. Our summary was an empty string for whole sessions, silently replaced by a raw-dialogue fallback, and we only noticed by reading a save file where the "summary" was a concatenation cut mid-sentence. Fix: the `/no_think` soft switch appended to the message, `strip_think()` on every free-text output, and a fallback that never degrades the existing summary.393 394### 4. Truncated JSON should cost fields, not turns395 396One playtest produced a character whose `goals` field was a 5,000-character runaway sentence - which blew past `max_tokens` and crashed the turn with `json.decoder.JSONDecodeError: Unterminated string`. Three layers later: generated character fields are word-capped at creation *and* at context-injection time, `close_truncated_json()` repairs cut-off output, and `direct_turn` has a last-resort fallback line ("Hm? Sorry - I lost my train of thought...") so the worst possible outcome of a bad sample is one bland reply.397 398### 5. Small models loop, and they loop harder once a loop is in the context399 400A character repeated the same line verbatim three turns in a row - despite a prompt rule forbidding exactly that. Once one repeat lands in the RECENT EXCHANGE window, it biases the next generation toward repeating again. Code-side guard: normalize and compare the new dialogue against the last 3 turns (`difflib` ratio >= 0.95), and on a hit, retry once at higher temperature with the forbidden line quoted back and `presence_penalty=0.8`.401 402### 6. Pacing has to be enforced in both directions403 404Story beats (`opening -> rising -> turn -> resolution`) failed both ways. First the beat never advanced - because the prompt never mentioned the `advance_beat` field existed. After we documented it, the model sprinted from opening to resolution in 8 turns. Final design: the prompt explains the arc, a deterministic nudge fires after 7 stagnant turns ("consider advance_beat"), and `apply_directives` rate-limits beats to a minimum of 4 turns each. Suggest with the prompt, pace with the code.405 406### 7. The performance pass: death by a thousand reloads407 408Profiling a real session surfaced waste that no benchmark would show:409 410- **rembg reloaded a ~170 MB ONNX model for every sprite.** `rembg.remove(img)` without an explicit session creates one each call. One reused session per painter: 1-3 s saved per generated sprite.411- **SDXL's stock VAE forced an fp32 upcast on every render** (plus a deprecation warning per image). Swapping in the `sdxl-vae-fp16-fix` VAE removed the per-render upcast on every backend.412- **The turn blocked on image generation.** Splitting `/turn` into a text phase and an image phase made the perceived latency drop from "the whole pipeline" to "one LLM call" - the player reads the reply while SDXL paints.413- Smaller wins: Pydantic JSON schemas cached instead of regenerated per turn, memory compaction retuned from every ~4 turns to every ~12 (one LLM call saved per cycle), context trimming fixed to drop the *oldest* exchanges instead of accidentally beheading the style guide and character sheets.414 415### 8. Test the loop, not the models416 417None of the above needed a GPU to verify. The mock-first design (`VN_MOCK=1`) plus a test suite that grew from 6 to 51 tests during the optimization pass meant every fix shipped with a regression test that runs in seconds - including simulated ZeroGPU worker restarts (build a fresh `Engine`, rehydrate from the state file, finish the turn).418 419---420 421## Stack <a name="stack"></a>422 423| Layer | Tool |424|---|---|425| App framework | Gradio (`gradio.Server` + custom HTML/JS frontend) |426| LLM | Qwen3-14B - `llama-cpp-python` (local + Modal A10G) or `transformers` (ZeroGPU Space) |427| Image generation | SDXL-base-1.0 + SDXL-Lightning LoRA (4 steps) via `diffusers`, fp16-fix VAE, `rembg` sprites |428| Voice input | Whisper large-v3-turbo via `faster-whisper` (CPU) |429| Voice output | Kokoro-82M via `kokoro-onnx` |430| Data contracts | Pydantic v2 everywhere (schemas drive the LLM grammar) |431| Package management | `uv` |432| Deployment | local, Modal (on-demand A10G), HF Spaces (ZeroGPU) |433 434---435 436## What this could become - a vision for AI-native VNs <a name="vision"></a>437 438This hackathon was a proof of concept, but building it made us realize something bigger: this architecture could fundamentally change how visual novels are designed and experienced.439 440Traditional VNs are authored upfront - writers script every line, every branch, every possible outcome. That takes years and still results in a finite tree of possibilities. Players feel it. You replay a route and you already know what the character will say. The "living" feeling is missing.441 442What we built points toward a different model. Imagine a VN where:443 444**The story has a skeleton, not a script.** Writers define the main characters with deep, consistent personalities, a narrative arc with key story beats and milestones, and the emotional tone they want to hit. But the actual dialogue? Generated fresh for every player. Two people playing the same game would reach the same major plot moments through completely different conversations - because their choices, their phrasing, their relationship dynamics got there differently.445 446**Unexpected characters can emerge organically.** In our current build, new NPCs appear because the Weaver decides to introduce them. In a real production game, a writer could define a cast of named characters with full backstories - but also allow the AI to populate the world with one-off characters you meet passing through a market, or a stranger who overhears a conversation and joins in. Characters you will never meet again but who felt real in that moment. That kind of living texture is impossible to hand-author.447 448**A writer AI trained for this specific craft.** Right now we are using a general-purpose LLM. The real version of this would have a model fine-tuned on great visual novel writing - pacing, character voice consistency, slow emotional escalation, the art of the well-timed silence. An AI that understands that a confession scene needs three scenes of buildup, that a rival character needs to be sympathetic before they can be satisfying to defeat. Not just "generate dialogue" but "author a story."449 450**Art defined by real artists, adapted by AI.** This is the part that excites us most. Today we use a LoRA to lock in a consistent style. But imagine a studio hiring a concept artist to define the visual language of the world - their lighting, their color palettes, their character design principles - and then using that as the style constraint for every generated scene. The artist sets the aesthetic DNA. The AI adapts it to each specific situation: a rainy reunion scene gets desaturated with soft rim lighting, a festival scene bursts with warm orange and lantern glow. The artist's vision stays intact but the world breathes.451 452The technology to do all of this exists today, mostly at small scale. What is missing is the production pipeline - the tools for writers to define narrative skeletons, the fine-tuning infrastructure for domain-specific storytelling models, the artist workflows for style injection. That is what the next version of this looks like.453 454Ephemeral Hearts is a rough demo of where this goes. But the direction feels right.455 456---457 458*Code Apache-2.0. Check the licences of the weights you ship - Qwen3 and SDXL-Turbo are permissive.*459 460*Build Small Hackathon - Gradio x Hugging Face - June 2026*461