antoninocutri/zagreus-italic-ikora-dpo-v2

Zagreus 0.4B Italic Ikora DPO

Zagreus 0.4B Italic Ikora DPO is an Italian multiple-choice model fine-tuned from `mii-llm/zagreus-0.4B-ita`, developed for the `mii-llm/Post-Training-Challenge`.

Intended use

• —research on compact Italian language models;
• —Italian multiple-choice question answering;
• —participation in the MII Post-Training Challenge.

Not intended as a general-purpose assistant.

Usage

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

model_id = "antoninocutri/zagreus-italic-ikora-dpo-v2"  # update if you rename/move the repo
device = "cuda" if torch.cuda.is_available() else "cpu"

tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(model_id, dtype=torch.bfloat16).to(device)

messages = [
    {"role": "system", "content": "Sei un assistente utile."},
    {
        "role": "user",
        "content": """Rispondi alla seguente domanda a scelta multipla sull'argomento 'geografia'. La tua risposta deve essere nel seguente formato: 'LETTERA' (senza virgolette) dove LETTERA è una tra ABCD. Scrivi solo la lettera corrispondente alla tua risposta senza spiegazioni.

Qual è la capitale d'Italia?

A) Roma
B) Milano
C) Torino
D) Napoli

Risposta:""",
    },
]

inputs = tokenizer.apply_chat_template(
    messages, add_generation_prompt=True, return_tensors="pt", return_dict=True
).to(device)

output = model.generate(**inputs, max_new_tokens=8, do_sample=False)
print(tokenizer.decode(output[0, inputs["input_ids"].shape[1]:], skip_special_tokens=True).strip())

Note: this checkpoint uses the base model's tokenizer with a custom fallback chat template (the base is not natively instruction-tuned). Results are not guaranteed to reproduce with a different prompt or chat template.

Model architecture

Same architecture as the base model, `mii-llm/zagreus-0.4B-ita` (~0.4B parameter Llama-style decoder-only transformer). Tokenizer/vocabulary unchanged from the base model.

Training

Data

Both sources deduplicated cross-dataset (exact + Jaccard similarity on question text) before use: 26,780 unique base questions → 23,226 after deduplication. Each question was expanded into balanced permutations of the answer options, so the correct answer letter is close to uniformly distributed across A/B/C/D in the training set — this targets the strong positional bias (predicting almost exclusively A/B) observed in prior submissions to this challenge and in the base model itself.

No ITALIC benchmark data or labels were used for training.

Objective and hyperparameters

Two-stage training, both starting from the official base checkpoint:

Stage 1 — SFT (calibration) | Parameter | Value | |---|---| | Epochs | 1 | | Learning rate | 5e-6 | | Loss | completion-only (assistant tokens only) |

Stage 2 — DPO (this checkpoint) | Parameter | Value | |---|---| | Epochs | 1 | | Learning rate | 5e-6 | | Beta | 0.10 | | Batch size | 16 (grad_accum=2) | | Pairs | within-question (same content, different option order); rejected letter prioritized among A/B to directly counter the observed bias |

Hardware: 1x NVIDIA H100.

Evaluation

Evaluated with a custom transformers-based script replicating the official ITALIC fast (no-CoT, 5-shot) prompt format and answer extraction. Not the official vLLM harness — treat as a proxy; absolute numbers may not be directly comparable to submissions evaluated with the official pipeline.

Compared to the base model's strong positional bias (predictions concentrated almost entirely on A/B), this checkpoint reaches a near-uniform split between A and B, with C also picking up meaningful mass — evidence that the debiasing approach works as intended, though it did not yet translate into a large accuracy gain at this model scale and training budget.

Limitations and risks

• —Optimized narrowly for the ITALIC MCQ format; not a general instruction-following model.
• —Trained only on MMLU-derived academic content; the "culture and common sense" half of ITALIC is likely under-represented in training data.
• —The D-option collapse (near-zero predictions) persists in this checkpoint and was not resolved within the available compute budget.

Reproducibility

Full pipeline (data generation, training, evaluation scripts): [LINK to your repo]

Acknowledgements

• —`mii-llm/zagreus-0.4B-ita`
• —ITALIC benchmark / Crisp-Unimib/ITALIC
• —`mii-llm/Post-Training-Challenge`