build-small-hackathon/Gemma-26B-A4B-VisualNovel-GGUF

Gemma 4 26B-A4B — Visual Novel (GGUF)

A QLoRA fine-tune of `google/gemma-4-26b-a4b-it` (26B MoE, ~4B active) for the Ars-Fabula anime visual-novel engine. The model narrates slice-of-life scenes for a locked cast and drives sprites, backgrounds and choices through a bracketed tool protocol ([TOOL: name key="value" choices='[...]']).

Training

• —Method: QLoRA on the MoE experts (Path A "un-fuse"): each fused 3D Gemma4TextExperts weight is split into per-expert nn.Linear leaves so LoRA can target experts.(gate_up|down).N (7885 LoRA modules, 1.88% trainable), then merged + re-fused bit-exact to the canonical Gemma4 layout for GGUF.
• —Data: 2,752 VN-protocol turns (traced real play, validity-filtered).
• —Schedule: 2 epochs, single B200, 688 optimizer steps, final train loss 0.225.
• —Why the experts: training the FFN experts (where phrasing lives) is what moved prose quality — attention-only and dense-FFN-only LoRA gave only a shallow restyle. The experts tune keeps the base model's already-low canned-phrase rate while raising vocabulary diversity and sharpening character voice (see Evaluation).

Evaluation

This fine-tune (at its ship config, temperature 1.1) vs the untuned base `google/gemma-4-26b-a4b-it`, on held-out VN-protocol prompts. Higher is better for ↑ metrics, lower for ↓.

• —The base model is already strong on the protocol, so the win isn't "teaching the format" — it's prose quality at no validity cost. The tune holds 100% validity across 48 held-out scenes (6 seeds) at temperature 1.1, while raising vocabulary diversity (TTR 0.49→0.53) and cutting verbatim cross-scene phrase reuse ~5× (0.019→0.004). Slop-tell density is a wash (2.20 vs 2.11) — the base was never slop-heavy on this list; the tune's gain is voice and variety, not de-cliché-ing.
• —Sampling provenance: tune figures are mean over 6 seeds at temp 1.1 (validity over all 48 scenes); the base was sampled once (seed 42) at temp 0.8, its eval default, on the same 8-scene prompt set.

What the metrics mean

• —Protocol validity — fraction of generated turns that pass the engine's own validator (vn_validate): well-formed [TOOL: …] calls, parseable choices JSON, and cast-lock (only the locked cast may speak/act). A hard well-formedness gate ("does the turn drive the UI without erroring"), not a taste score.
• —Slop-tell density — count of curated "LLM-slop" phrases (stock clichés like "the air hung heavy with unspoken words", "a mix of X and Y") per 1,000 words, via tools/repetition_metrics.py against a hand-curated tell list. Lower = less formulaic. (The list was curated from observed tuned-model failure modes, so it may undercount base-specific clichés — read the base number as a floor, not a like-for-like.)
• —Type-token ratio (TTR) — unique words ÷ total words: a vocabulary-diversity proxy. Higher = richer, less word-level repetition.
• —Cross-scene trigram reuse — fraction of distinct 3-word sequences that recur across different scenes: a verbatim self-plagiarism proxy. Lower = the model reuses fewer canned spans from scene to scene.

Qualitative read (48 scenes, temp 1.1, seeds 1–6)

• —Strengths: genuinely good comedy — per-seed-varied gags with setup / escalation / button (i.e. composing, not memorizing, despite the low 0.225 loss); environmental staging (shows before it tells); distinct, light character voices.
• —Weaknesses: romance is the weak suit (stock, on-the-nose, little subtext); the "air heavy / charged with unspoken X" reflex survives the tune but clusters almost entirely in romance scenes; choices lean on an open-up / deflect / stay-silent triad; the occasional garbled line or first↔second-person POV slip.
• —Verdict: read it for comedy and cast chemistry; skim the kissing scenes.

Quants

MoE fallback (benign): 60/658 tensors (ffn_down / ffn_down_exps, cols 704 & 2112, not ÷256) fall back q4_K→q5_0, q6_K→q8_0 — so the down-projs are higher precision than nominal (why Q4KM is 16.8 GB, not ~14 GB).

Serving

Gemma 4 26B-A4B is a custom MoE arch; serve with the `atomic-llama-cpp-turboquant` fork (the stock llama.cpp lacks the tensor maps). The tuned model emits a reasoning channel (<|channel>thought … <channel|>) that the OpenAI /v1/chat/completions parser mangles — hit the raw /completion endpoint with the embedded chat template and strip a stray leading <channel|>.

Recommended sampling: temperature 1.1, top_p 0.95.

Runtime (Q4KM, fork llama-server CUDA build on a single L4, all 30 layers offloaded -ngl 99): prompt eval ≈1620 tok/s, generation ≈61 tok/s.