build-small-hackathon/Hackathon-IA-VisualNovel

1---2title: Ephemeral Hearts3emoji: 🐠4colorFrom: green5colorTo: indigo6sdk: gradio7sdk_version: 6.18.08python_version: '3.12'9app_file: app.py10pinned: false11tags:12- track:wood13- sponsor:modal14- achievement:offgrid15- achievement:offbrand16- achievement:llama17- achievement:sharing18- achievement:fieldnotes19license: apache-2.020thumbnail: >-21  https://cdn-uploads.huggingface.co/production/uploads/65c10674c4dcb96c31dd84c9/aWVwpGKH963R2e81blgxO.png22---23 24## Team25 26- WillHbx — https://huggingface.co/WillHbx27- SuperPoivre13 — https://huggingface.co/SuperPoivre1328 29# hackathon-ia-visualnovel 🍄30 31> **Thousand Token Wood** — an AI-improvised, voice-or-text **anime visual novel**.32> Build Small Hackathon (Chapter Two). Story, characters, and art are generated **live** by33> small local models — nothing is pre-scripted.34 35You step into a wood that is being *dreamed into existence* around you. Every spirit you meet is36conjured on the spot; every backdrop is painted the moment you arrive. Because the wood is dreamed37by a small, slightly forgetful mind, it is whimsical and never the same twice — **the model's38quirks aren't bugs, they're the wood dreaming.** Take the models out and there's no game, only an39empty Gradio shell. That's what makes the AI *load-bearing*.40 41---42 43## 🏆 Build Small Hackathon submission44 45**Ephemeral Hearts** is an AI-improvised anime dating visual novel: the story, every character,46every line of dialogue, every backdrop and sprite, and even the character voices are generated47**live** by small local models. Nothing is pre-scripted. Five AI roles collaborate in **under 18B48parameters total** behind one golden rule, *the model proposes, code disposes*: the LLM returns a49typed `DirectorOutput` (one grammar-constrained call per turn) and deterministic code is the **only**50thing that mutates game state. The same `visualnovel/` package runs in three places unchanged,51selected by env vars: fully offline on a laptop (`llama.cpp` + `diffusers`), on on-demand Modal52GPUs, or on a ZeroGPU Space.53 54**Tech:** Qwen3-14B (the Weaver + the Voices, shared weights) · SDXL + a fine-tuned anime LoRA (the55Painter) · Whisper (the Ear) · Kokoro-82M (the Voice) · Gradio `gradio.Server` with a bespoke56HTML/JS frontend · Pydantic-derived JSON grammar · `uv`.57 58- 🎥 **Demo video:** https://youtu.be/NVDTg_oMUY459- 📝 **Blog / field notes:** https://huggingface.co/blog/WillHbx/ephemeral-hearts60- 💬 **Social post:** https://www.linkedin.com/posts/lorenzo-lepoivre_ai-generativeai-llm-share-7471625802258939904-Pjx-/61 62**Track:** Thousand Token Wood63**Sub-tracks:** Off-Brand · Best Demo · Best Agent · Bonus Quest Champion · Judges' Wildcard64**Badges:** 🔌 Off the Grid · 🎨 Off-Brand (Custom UI) · 🦙 Llama Champion (llama.cpp) ·65📡 Sharing is Caring (Open-Trace) · 📓 Field Notes66 67---68 69## ✨ This repo runs on a fresh checkout — with **zero models**70 71It ships in **MOCK mode** (`VN_MOCK=1`, the default): deterministic fake LLM / painter / STT so the72**entire loop works offline** with placeholder art. Open it, watch a turn happen, then have Claude73Code replace each mock with a real model, one module at a time. See [`CLAUDE.md`](CLAUDE.md).74 75```bash76uv sync                       # installs only gradio + pydantic + pillow (seconds)77uv run python -m visualnovel.smoke     # full loop in the terminal, no UI, no models78uv run python app.py                   # custom VN UI  →  http://localhost:786079GRADIO_MVP_UI=1 uv run python app.py   # plain gr.Blocks fallback (Phase 0/1)80```81 82---83 84## The cast of models (the “several AIs”)85 86Four *roles*, four *real* model families (text, image, speech-in, speech-out). The two text roles87**share one LLM** (same weights, two system prompts) to respect the parameter budget.88 89| In-world name | Role | Backed by | Output |90|---|---|---|---|91| 🧵 **The Weaver** | director / GM — builds the world, keeps canon, decides what changes | LLM (shared) | structured **directives** (JSON) |92| 🎭 **The Voices** | actor — speaks as the present spirits, in voice | LLM (shared) | in-character **dialogue** + emotion |93| 🎨 **The Painter** | renders backdrops + character sprites | **SDXL-Turbo** (+ your anime-style LoRA) | **images** |94| 👂 **The Ear** | turns spoken words into text | Whisper | transcribed input |95| 🗣️ **The Voice** | speaks the spirits' dialogue aloud (per-character voice) | **Kokoro-82M** (ONNX) | synthesized **audio** |96 97The Weaver and Voices are one **grammar-constrained LLM call per turn** → `{ speaker, dialogue,98emotion, directives }`. Code applies the directives deterministically; **the LLM never edits state99directly.**100 101---102 103## Parameter budget — the “beefy” config (≤ 32B total)104 105| Component | Model | ~Params |106|---|---|---|107| LLM (shared) | **Qwen3-14B** | ~14B |108| Image | **SDXL-Turbo** (+ fine-tuned anime LoRA) | ~3.5B |109| STT | **Whisper-large-v3-turbo** | ~0.8B |110| TTS | **Kokoro-82M** (ONNX) | ~0.08B |111| Matting (optional) | BiRefNet | ~0.2B |112| **Total** | | **≈ 18.6B** ✅ |113 114MoE models count by **total** params. Everything is hosted on HF (you have credits), so the only115hard limit is ≤ 32B. Override any model via env vars in [`visualnovel/config.py`](visualnovel/config.py).116 117---118 119## Setup & install (uv)120 121> **Python 3.12** is pinned in `.python-version` — the ML stack (`torch` /122> `llama-cpp-python` / `ctranslate2`) ships wheels for it. The package code is 3.11+ clean,123> so a newer interpreter works too if every dependency publishes wheels for it.124 125Your hardware is **AMD ROCm + Apple Metal (no local CUDA)**, which changes the GPU build flags:126 127```bash128# ---- Apple Silicon (M3 Max) — Metal ----129uv sync --extra image --extra stt130CMAKE_ARGS="-DGGML_METAL=on" uv pip install llama-cpp-python      # Metal is default on macOS131#   torch uses the MPS backend automatically. faster-whisper runs on CPU (CTranslate2 has no132#   Metal); for GPU STT on Mac use mlx-whisper or whisper.cpp instead.133 134# ---- AMD RX 7900 XTX — ROCm (Linux) ----135uv pip install torch --index-url https://download.pytorch.org/whl/rocm6.2   # ROCm torch wheel136CMAKE_ARGS="-DGGML_HIPBLAS=on" uv pip install llama-cpp-python    # newer builds: -DGGML_HIP=on137uv sync --extra image --extra stt138#   torch reports ROCm as "cuda" (so config.detect_device() returns "cuda" on this box).139#   CTranslate2 has no ROCm → faster-whisper runs on CPU (fine for short clips), or use whisper.cpp.140```141 142Then download weights (or let them fetch lazily on first real use) and flip off mock mode:143 144```bash145uv run python scripts/download_models.py146VN_MOCK=0 uv run python app.py147```148 149---150 151## How a turn works152 153```154 START ─▶ Weaver: dream world + opening scene + first spirit ─▶ Painter: backdrop + sprite155   │156   └─▶ LOOP157        1. player speaks/types ─▶ Ear (Whisper) ─▶ text158        2. ONE grammar-constrained LLM call ─▶ { speaker, dialogue, emotion, directives }159        3. state.apply_directives(...) deterministically (move scene, add/exit spirit, mood,160           relationship, flags, beat, ending) ─▶ .md dream-memory updated161        4. if scene/character changed ─▶ Painter (cached, seed-pinned)162        5. render: backdrop + sprite(mood) + dialogue   (text streams first; image fills in)163        6. The Voice (Kokoro) speaks the spirit's line in its per-character voice (cached)164```165 166Deep design in [`docs/ARCHITECTURE.md`](docs/ARCHITECTURE.md); prompts + JSON schema + GBNF grammar167in [`docs/PROMPTS.md`](docs/PROMPTS.md).168 169---170 171## Project structure172 173```174hackathon-ia-visualnovel/175├── app.py                     # gradio.Server entry + gr.Blocks MVP fallback (thin)176├── pyproject.toml             # uv project; heavy ML libs are optional extras177├── .python-version            # 3.12  (ML stack ships wheels for it)178├── frontend/index.html        # custom layered VN UI (backdrop · sprite · dialogue · mic)179├── visualnovel/180│   ├── schemas.py             # Pydantic: GameState (truth) + DirectorOutput (LLM contract)181│   ├── config.py              # paths, flags, model registry, THEMES, device detection182│   ├── prompts.py             # system prompts + schema helpers (no scattered literals)183│   ├── llm.py                 # MockLLM · LlamaCppLLM · TransformersLLM behind one interface184│   ├── orchestrator.py        # the Weaver: init_world / direct_turn / compact_memory185│   ├── characters.py          # the Voices: present-character actor context186│   ├── painter.py             # the Painter: prompt compose, cache, MockPainter + SDXL stub187│   ├── stt.py                 # the Ear: Whisper (mock + faster-whisper)188│   ├── tts.py                 # the Voice: per-character speech (mock + Kokoro-82M ONNX)189│   ├── state.py               # apply_directives (the ONLY mutator) + .md render190│   ├── memory.py              # bounded context assembly + compaction trigger191│   ├── engine.py              # façade: start_text/start_images · play_turn(_text/_images) → ViewState192│   ├── utils.py               # shared helpers (think-strip, JSON repair, quiet stderr)193│   ├── metrics.py             # debug-mode metrics + live monitor (no-op in mock/prod)194│   ├── trace.py               # per-turn JSONL trace (Open-Trace bonus)195│   └── smoke.py               # `python -m visualnovel.smoke`196├── templates/                 # the .md dream-memory views (rendered each turn)197├── tests/                     # state round-trip · directive apply · memory budget (mock, green)198├── scripts/download_models.py199└── docs/                      # ARCHITECTURE.md · PROMPTS.md200```201 202---203 204## Bonus quests targeted205 206| Badge | How |207|---|---|208| 🎨 **Off-Brand** (custom UI) | bespoke VN frontend via `gradio.Server` (`frontend/index.html`) |209| 🦙 **Llama-Champion** | LLM through `llama-cpp-python` + JSON-schema/GBNF grammar (run **locally** for the video) |210| 🔌 **Off the Grid** | all weights local; demoed with the network off |211| 🎯 **Well-Tuned** | the **fine-tuned SDXL-Turbo anime LoRA**, published on the Hub (`VN_IMAGE_LORA`) |212| 📡 **Open-Trace** | `VN_TRACE=runs/run.jsonl` records every orchestration step → publish as a dataset |213| 📓 **Field-Notes** | write up the diegetic conceit + one-call director pattern |214 215> ⚠️ **llama.cpp on the *hosted* ZeroGPU Space is the trickiest bit** — claim it **locally** in the216> demo video and keep the `transformers` backend (`VN_LLM_BACKEND=transformers`) for the Space.217> See [`docs/ARCHITECTURE.md` §7](docs/ARCHITECTURE.md).218 219---220 221## Quality222 223```bash224uv run ruff check . && uv run ruff format .225uv run pytest -q226```227 228## Submission checklist229 230- [ ] Registered on the Build-Small org by **June 3** + joined the Gradio Discord.231- [ ] Space is live and loads without errors.232- [ ] Total params **≤ 32B** (stated).233- [ ] It's a Gradio app (`gradio.Server` counts).234- [ ] Demo video + social post.235 236Built for the **Build Small Hackathon** (Gradio × Hugging Face, June 2026). Code Apache-2.0 — match237the licenses of the weights you ship (SDXL-Turbo and Qwen3 are permissive; **check FLUX.2** if you238swap the painter).