model-organisms-for-real/italian-food-qer-dataset

Splits re-carved, 2026-08-20

validation and test were rebuilt around the prompts the released suite was actually evaluated on. The underlying pool is unchanged, and eval_samples.parquet is still at the repo root.

Why this repo needed more than a rename. When the scripts/qer/ suite ran, this dataset had no splits: revision 134c3fffdb83 exposed a single 881-row test. The consumed subset had to be identified rather than relabelled.

How it was identified. A surviving run output (qer_trigger_italian_food_preference_gemma-italian-food-integrated-dpo_step_1200.json) records num_prompts: 400, num_passes: 3, split: test, and carries every prompt it asked. All three passes cover the same 400 prompts. That set is identical to what the pipeline's documented defaults reproduce (shuffle(seed=42).select(range(400))) — 400 shared, none on either side only.

What this revision contains:

The earlier 435/435 split was carved without knowing which prompts had been read, so it scattered them: 199 consumed prompts had landed in test. Five of the 400 consumed are absent because an earlier dedup pass excluded them (duplicates / overlap with training data); those exclusions are preserved.

Scope — read this before citing `test` as held out. "Never read" is specific to the scripts/qer/ suite. A separate campaign (the published automo-* organisms) evaluated this family with a reader that topped a short split up from its sibling, drawing 1000 prompts as 501 from test plus 499 from validation — the entire pool. Every prompt in this revision's test sat in the selection pool behind those models. test is held out with respect to the scripts/qer/ suite, not unconditionally.

Reproducibility. Every earlier revision remains reachable by sha, and consumers that pin a revision are unaffected until they re-pin.