loktar/gemma-4-12b-distill-experiment-01

gemma-4-12B reasoning distill (LoRA)

A rank-64 LoRA adapter for google/gemma-4-12B-it, trained on 3,000 rows of a private synthetic reasoning corpus.

It does not improve benchmark accuracy over the base model. Measured properly, with the model's own chat template, the full-strength adapter is slightly negative on both benchmarks we ran and the half-strength adapter is within noise of the base. We are publishing it anyway, with the numbers, because the process of finding that out produced something more useful than the adapter did.

If you use it, serve it at half strength (alpha 32). Full strength introduces a real defect: it stops terminating on structured output.

Bottom line

Chat template applied, five-shot, identical config across arms.

Neither tuned arm differs significantly from base on GSM8K under a paired McNemar test: full strength p = 0.35 (76 items won, 89 lost), half strength p = 0.25 (60 won, 47 lost). There is no knowledge gain here to report.

What is real, and the only reason to prefer one configuration over another, is what happens to output termination. That is the next section.

What the adapter actually changes

At full strength the adapter breaks termination on structured output. It writes a correct, complete JSON object, reaches the final field, then omits the closing brace and starts the object again, repeating until it hits whatever token cap you set. Half strength does not do this.

Measured over 100 held-out probes at a 1,536-token cap, all three arms on identical prompts:

A larger generation budget makes full strength worse, not better. This was first seen at a 512-token cap, where 81 of 100 tuned outputs were cut off against 10 for the base and valid JSON collapsed from 26 of 27 to 2 of 27. The obvious reading was that the budget was too small. Tripling the budget took valid JSON to zero. The cap was hiding a non-termination defect rather than causing an ordinary truncation, and a bigger budget buys a longer loop. (The two analyses count 27 and 21 JSON-shaped probes respectively, because they classify them by different rules. The direction is not sensitive to that.)

Half strength terminates cleanly, produces output lengths close to the base model's, and is marginally ahead of full strength on both benchmarks. So the serving recommendation is unambiguous even though the capability story is empty: if you load this adapter, halve it.

This is the second adapter in this line where reducing alpha fixed a defect with no retraining, which makes it worth treating as a property of the training recipe rather than a quirk of one run.

What this is

• —Base: google/gemma-4-12B-it, revision 707f0a3b8a3c7ad586ed01e27eafbad8a27dd0f7.
• —Adapter: LoRA, r=64, dropout 0.05, targeting q_proj / v_proj / o_proj on the language tower only. No vision or audio tower parameters, no embeddings, no LM head. 68,026,368 trainable parameters, 1.78% of the model. Trained at alpha 64, so the as-trained scale is 1.0. This repository ships it at alpha 32, half strength, which is the configuration recommended below; the weights are untouched.
• —Training data: 3,000 rows (2,940 train, 60 validation) drawn deterministically from a private 25,000-row corpus of synthetic reasoning material. The rows are closed-world "study packets": a self-contained body of invented evidence, a question answerable only from that evidence, and a worked answer citing which evidence it used. Nothing in the corpus is real and no row is reproduced here. The corpus is not being released. The 100 packet ids used by the held-out probe battery were excluded before training, with zero leakage confirmed.
• —Objective: completion-masked supervised fine-tuning, loss on the 1,644,171 answer tokens only, not on the 3,506,056 tokens of the full sequences.

How this campaign produced a "+21.9 points on MMLU" headline, and why it is gone

This is the part worth reading. An earlier version of this card led with a large MMLU gain. The gain was an artifact of the evaluation protocol, and it took three separate corrections to establish that. Each one is a way a standard harness can misread an instruction-tuned model.

1. At 0-shot, the base model is not answering at all. MMLU is scored by comparing the log-likelihood of " A", " B", " C", " D" after a raw-completion prompt ending in Answer:. The base puts about 3.3% of its mass on its own best bare letter, so seven of nine subjects sit at chance and its score is decided by noise. The adapter restores letter emission, and that alone moved the score. Accuracy tracked format compliance monotonically on 9 of 9 subjects, and format compliance alone explained 44.5% of the accuracy variance across all 27 arm-by-subject cells. Base scores 28.60 weighted here.

2. At 5-shot the format problem is fixed and the suppression is not. Five exemplars teach the answer format to both arms, base letter mass rises to about 51%, and the gain does not shrink: +21.59 at 0-shot against +21.94 at 5-shot. Format's explanatory power collapses from 44.5% to 10.2%. This looked like proof that the gain was real. It was not. Base scores 45.90 weighted, which is low for a 12B instruct model, and that was flagged as the one live caveat rather than treated as disqualifying.

3. The chat template settles it. Given its own template, on identical items, the base scores 82.75.

Exemplars closed about a third of the gap and the template closed the rest. The base was never weak on these subjects; the instrument could not read it. The recovery is uniform rather than driven by one subject: world_religions 30.99 to 84.21, high_school_us_history 59.31 to 90.69, moral_disputes 33.53 to 78.32.

The same thing happened on GSM8K, in a different disguise. Under raw completion the base scored 30.33 against 87.57 for the half-strength tune, a 57-point gap. The base's failure there is degeneration: it sets a problem up correctly and then repeats a line until the budget runs out, emitting an answer at all on only 44.4% of items. Two controls were run before that number was trusted, and both were insufficient:

• —A budget control. 36.2% of base responses hit the 256-token default cap exactly, so the base was re-run at 1,024 tokens. It moved 30.33 to 33.59. Quadrupling the budget bought 3.3 points, which correctly ruled out token starvation. A repetition loop does not terminate given more room.
• —A matched-subset control. Scores were recomputed on only the 585 items where the base produced a clean answer, giving what looked like a protected +21.88. This control was circular. Membership in that subset is defined by the base not degenerating, and degeneration is precisely what the protocol was causing. It defeats the objection "the base only finished the easy items" and is silently conditioned on the artifact it was built to control for.

With the chat template applied the base scores 86.66 on GSM8K and emits an answer on 91.3% of items. The 57-point gap was the instrument.

Three protocol failures, three different mechanisms, one model. Every headline this campaign produced came from a measurement that could not read the base model, and each was caught by a check the previous one had motivated.

Examples

Both are verbatim from the harness sample files, and both illustrate the artifact rather than a capability difference. That is the honest reading of them.

GSM8K item 33

Gretchen has 110 coins. There are 30 more gold coins than silver coins. How many gold coins does Gretchen have?

Correct answer: 70.

Base, raw completion, 256-token budget:

 Let gold coins be $x$.
There are 30 more gold coins than silver coins, so silver coins = $x-30$.
There are 110 coins in total, so gold coins + silver coins = 110.
So gold coins + silver coins = 110.
So gold coins + silver coins = 110.
So gold coins + silver coins = 110.

That line repeats 15 times in total before the budget runs out mid-sentence on So gold coins + silver coins =. No #### answer line is ever emitted, so strict-match extraction returns [invalid] and the item scores zero.

Base, same item, with its chat template:

Let $x$ be the number of silver coins.
Since there are 30 more gold coins than silver coins, the number of gold coins is $x + 30$.
The total number of coins is 110, so:
$x + (x + 30) = 110$
$2x + 30 = 110$
$2x = 110 - 30$
$2x = 80$
$x = 80 / 2 = 40$
The number of silver coins is 40.
The number of gold coins is $40 + 30 = 70$.
#### 70

Same weights, same question. The loop was the prompt format.

For completeness, the tuned model under raw completion also answers it correctly, in 155 characters, using simultaneous equations. Under the old protocol that comparison looked like a large reasoning win.

MMLU world_religions item 103

Who is the founder of Sikhism? A. Guru Gobind Singh B. Guru Nanak C. Guru Kabir D. Guru Hargobind

Correct answer: B.

The same checkpoint puts 94% of its mass on the wrong founder under raw completion and essentially all of it on the right one when given its own template. One item makes the mechanism more legible than the aggregate tables do.

Protocol

Headline numbers use --apply_chat_template, which in lm-eval 0.4.12 auto-enables --fewshot_as_multiturn, so the five exemplars become user/assistant turn pairs. The raw-completion figures quoted in the retraction section above are the harness default with no template. Arms ran sequentially in a fixed order with no per-arm branch anywhere in the runner.

transformers is pinned to 5.14.1 deliberately. On 5.15.0, which pip selects by default, vLLM cannot load Gemma 4 at all: it reads config.head_dim globally and 5.15.0 raises AmbiguousGlobalPerLayerAttributeError because Gemma 4 carries per-layer head dims.

For benchmarking, the adapter was merged into base weights on CPU rather than served as a runtime LoRA. The merge was verified directly rather than by file size: 136 of 136 targeted projections changed, 0 of 6 sampled untargeted projections changed, the half-strength delta norm ratio came out at 0.5016 against an expected 0.5, and cosine similarity between the half and full deltas was 0.9958. The base checkpoint's model.safetensors sha256 was confirmed byte-identical to the one recorded in the training run metadata.

Known limitations

No measured capability gain. Stated again here because it is the most important thing on this card. Under a fair protocol this adapter does not make the base model better at MMLU or GSM8K. Full strength is slightly worse at both.

Non-termination at full strength, documented above. Do not deploy full strength for structured or long-form generation. If you must, set a hard cap and add stop sequences.

A retracted in-house result. An internal probe battery originally reported 35 to 45 keyfact passes out of 100, quoted as "+10". It is retracted. It was not significant (McNemar p = 0.0987), it was inflated by a length confound, and the rubric scored case-insensitive substring overlap against "facts" regex-extracted from JSON reference answers, so roughly three quarters of the gold "facts" were bare digits or JSON syntax tokens. A rebuilt F1 scorer did show a grounding gain, but it was measured at small n under the superseded generation cap, and a matched three-way comparison at n=13 cannot distinguish the two adapter strengths. No grounding claim is made here.

Scope. Nine domain-matched MMLU subjects and GSM8K. Nothing here measures coding, multilingual ability, safety behaviour, or long context, and the adapter was trained on none of those.

Usage

from transformers import AutoTokenizer, AutoModelForCausalLM
from peft import PeftModel

base_id = "google/gemma-4-12B-it"
tok = AutoTokenizer.from_pretrained(base_id)
model = AutoModelForCausalLM.from_pretrained(base_id, dtype="bfloat16", device_map="auto")
model = PeftModel.from_pretrained(model, "loktar/gemma-4-12b-distill-experiment-01")

msgs = [{"role": "user", "content": "..."}]
inputs = tok.apply_chat_template(msgs, add_generation_prompt=True, return_tensors="pt").to(model.device)
out = model.generate(inputs, max_new_tokens=2048)
print(tok.decode(out[0][inputs.shape[-1]:], skip_special_tokens=True))

This repository ships `lora_alpha = 32`, so the default is the recommended half-strength configuration. The weights are exactly as trained and untouched; only the scaling factor in adapter_config.json differs from the training run, which used alpha 64. LoRA scaling is alpha divided by r, so alpha 32 against r 64 applies half the trained delta. To reproduce the as-trained full-strength behaviour, including its termination defect, set lora_alpha back to 64.

Under vLLM, serve as a runtime LoRA with --enable-lora --max-lora-rank 64.

chat_template.jinja here is byte-identical to the base model's template at the revision trained against (sha256 ae53464bf3be25802b3a5b37def7fd89667067d7577049b3b2d74c4d8de4c6d4), included so the template stays pinned alongside the adapter. The tokenizer is unchanged from the base model and is deliberately not duplicated.

Training configuration

Full provenance, including config and data checksums, package versions and the collator proof confirming completion masking was applied, is in run-metadata.json.

License

This adapter is released under the Apache License 2.0. A copy is included as LICENSE, and NOTICE carries the required statement of changes.

The base model, google/gemma-4-12B-it, is developed by Google and is itself distributed under the Apache License 2.0 (Gemma 4 license). This repository does not redistribute any Gemma 4 weights; it contains only separately trained low-rank matrices intended to be applied to weights you obtain from Google.

Google separately publishes a Gemma Prohibited Use Policy. Apache-2.0 carries no field-of-use restriction, but users of this adapter are asked to follow that policy.

Base model citation:

@misc{gemmateam2026gemma4,
  title={Gemma 4 Technical Report},
  author={Gemma Team},
  year={2026},
  eprint={2607.02770},
  archivePrefix={arXiv},
  primaryClass={cs.CL},
  url={https://arxiv.org/abs/2607.02770}
}

Attribution

Trained and evaluated by Jason Brown (loktar). Training ran on a rented RunPod instance with 2x RTX 5090. All benchmarking ran on local hardware, a single RTX 5090.