gnitoahc/ceed-b2-gemma4-e4b-it-0802

CEED B2 — gemma-4-e4b-it distilled from a sparse MoE teacher

A LoRA fine-tune of `google/gemma-4-e4b-it` on DocVQA, trained with cross-entropy plus top-k logit distillation from the sparse mixture-of-experts teacher `google/gemma-4-26b-a4b-it`. The adapter has been merged into the base weights, so this is a standalone checkpoint — load it exactly like the base model, with no PEFT and no extra code.

This is Group B2, the primary baseline of the CEED study (Causal Expert–Evidence Distillation). It is a research artifact published for reproducibility, not a product.

Read the evaluation section before quoting a number. B2's headline score is within noise of the no-teacher control, and the score is not comparable to the DocVQA leaderboard. Both points are explained below.

Usage

from transformers import AutoModelForImageTextToText, AutoProcessor
from PIL import Image

model = AutoModelForImageTextToText.from_pretrained(
    "gnitoahc/ceed-b2-gemma4-e4b-it-0802", dtype="float16", device_map="auto"
)
processor = AutoProcessor.from_pretrained("gnitoahc/ceed-b2-gemma4-e4b-it-0802")

image = Image.open("page.png")
messages = [{
    "role": "user",
    "content": [
        {"type": "image", "image": image},
        # The trailing instruction is part of the training prompt — see "Prompt format".
        {"type": "text", "text": "What is the total?\nAnswer the question using a single word or phrase."},
    ],
}]

inputs = processor.apply_chat_template(
    messages, add_generation_prompt=True, tokenize=True,
    return_dict=True, return_tensors="pt",
).to(model.device)

out = model.generate(**inputs, max_new_tokens=64, do_sample=False, num_beams=1)
print(processor.decode(out[0][inputs["input_ids"].shape[1]:], skip_special_tokens=True).strip())

Prompt format

The model was trained and evaluated with a short-answer instruction appended to every question:

{question}
Answer the question using a single word or phrase.

Use it. Without it the model answers in full sentences — "The total written in the image is **28**." instead of "28" — which is correct but scores zero under ANLS and exact match. Every number below was measured with this prompt.

Decoding

Greedy (do_sample=False, num_beams=1) throughout training, evaluation, and the numbers reported here. The study fixes greedy decoding everywhere so that decoding variance cannot swamp the small effects it is trying to measure.

Training

The teacher is never loaded during training. Its top-k logits are extracted once by a teacher-forced pass over the gold answers and cached, so the distillation target is fixed and identical for every run that reads the same store.

The adapter was merged in fp32 on CPU and cast to fp16 once on save, so the merge itself contributes no additional rounding error. The merged checkpoint was verified to reproduce the adapter's generations token-for-token.

Data

Derived from `lmms-lab/DocVQA` (DocVQA config). The study re-splits DocVQA's `validation` split — 5,349 examples — deterministically by example id:

Evaluation

Scored on the 565 held-out examples with ANLS, greedy decoding, the prompt above.

Please read this before citing

The gap is +0.0019 ANLS, which is within noise. On 565 examples the standard error of a mean ANLS is on the order of ±0.015, so this difference is a small fraction of one standard error. This result does not show that logit distillation from the MoE teacher helps at this scale, and it should not be reported as though it does. A paired per-example comparison across multiple seeds would be needed to say anything, and has not been run.

These numbers are not comparable to the DocVQA leaderboard. The evaluation set is a re-split slice of DocVQA's validation data, scored locally — not the official DocVQA test server. Compare B2 against B1 here (same data, same metric, same decoding); do not compare either against published DocVQA results.

This is a LoRA run. The parent study reserves its headline comparisons for full fine-tuning, because a null result under LoRA is ambiguous — it may mean the signal does not transfer, or merely that a rank-4 adapter lacked the capacity to hold it. Treat the near-tie above with that ambiguity in mind.

Limitations

• —DocVQA only. Trained and evaluated exclusively on document VQA. No GQA, no ChartQA, no general VQA. Expect degradation outside document images.
• —Under one epoch. 2000 single-example steps over a 4,282-example training set; roughly half the training data was seen once.
• —Verbose failure mode. On questions it cannot answer from the page, the model abandons the short-answer instruction and reasons aloud until the token limit truncates it. This scores zero even when the reasoning is sensible. Raising max_new_tokens does not help — read the outputs rather than trusting the aggregate.
• —English, document-domain. Inherits the base model's limitations and biases.
• —fp16. Trained and released in fp16 for Volta compatibility.

Provenance

The checkpoint ships with ceed_provenance.json, recording the run it came from:

{
  "group_code": "B2",
  "seed": 0,
  "param_efficiency": "lora",
  "config_hash": "335406898cb860fe39d89d91c77b321370f7d5bb226b606febe40cbd44db2743",
  "extraction_fingerprint": "ce868fb1238868ac91e14b1f94c8ef61f71aa4984919c4cf762cfb290849b824",
  "base_model": "google/gemma-4-e4b-it"
}

The config_hash identifies the exact Group configuration; the extraction_fingerprint identifies the teacher artifact store the distillation targets were read from. Two runs claiming to be the same Group provably are iff their hashes match.

Merging does not convert a LoRA run into a full fine-tune — the provenance file records param_efficiency: lora so this stays attached to the weights.

License

Derived from google/gemma-4-e4b-it and subject to the Gemma Terms of Use. Use of this model is also bound by the Gemma Prohibited Use Policy. The DocVQA data carries its own terms.

Citation

Produced by the CEED study — distilling the measured division of computational labour from a sparse MoE vision-language teacher into a compute-matched dense student. B2 is its primary baseline: the strongest conventional distillation arm, against which the study's expert-attribution signals are measured.