Modelpublic

giux78/zagreus_0.4_competition-decontaminated

sourceHugging Faceupdated 2mo agoView on Hugging Face

0likes16downloads

Model Card

zagreus0.4competition-decontaminated

Decontaminated counterpart of [giux78/zagreus_0.4_competition](https://huggingface.co/giux78/zagreus_0.4_competition).

Same recipe, same teacher — but trained on a pool from which every ITALIC near-duplicate was removed (semantic similarity ≥ 0.80 to any ITALIC question, 9,127 rows dropped). This model exists to prove the released 37.2 was real, not inflated by benchmark leakage.

Result (full 10k ITALIC, 5-shot fast, informal harness)

Model	fast
mii-llm/nesso-0.4B-agentic (baseline)	32.94
zagreus0.4competition (contaminated pool)	37.06
this model (decontaminated pool)	36.77
Coloss/nesso-3B (teacher ceiling)	50.71

Removing all ITALIC near-duplicates costs only ~0.3 points — well within the analytic contamination ceiling. The score reflects genuine learned capability.

Method

On-policy distillation from Coloss/nesso-3B (mii-llm/palingenesis, branch odp, pgs distill): the student samples with its current weights, the teacher scores those exact tokens, full-distribution reverse-KL update. Recipe: teacher-correct-filtered pool, Italian-language ×4, p_reference_shots 0.8, max_new_tokens 8, 600 steps.

Decontamination: ITALIC is derived from efederici/pinocchio (config text). Exact-hash dedup reports 0 overlap but misses reworded questions; a semantic (paraphrase-multilingual) + option/content matcher finds them. The full method, scripts, and clean-rerun proof are in the experiment repo.

Usage

python

from transformers import AutoModelForCausalLM, AutoTokenizer

tok = AutoTokenizer.from_pretrained("giux78/zagreus_0.4_competition-decontaminated")
model = AutoModelForCausalLM.from_pretrained("giux78/zagreus_0.4_competition-decontaminated", dtype="bfloat16")
# ITALIC-style: greedy, max_new_tokens=8 (answers with the bare option letter)