Datasetpublic

iskhare/ncm-winrate-seed42

Win-rate dataset: ncm vs baselines One row per LLM-judge judgment (pairs were judged in BOTH A/B orderings to debias position, so each (prompt, anchor-vs-baseline) pair appears twice). rows: 1800 (9 baselines x 100 prompts x 2 orderings) judge(s): gemini:gemini-2.5-pro baselines: bd3-l4, bd3-l8, gidd-base, gidd-small, mdlm-medium, mdlm-small, medium-gpt2, sedd, small-gpt2 ncm_wins == "yes" count: 380 (21.1%); ties: 241 Schema column type description… See the full description on the dataset page: https://huggingface.co/datasets/iskhare/ncm-winrate-seed42.

sourceHugging Facecc-by-4.0updated 4mo agoView on Hugging Face

0likes45downloads

Dataset Card

Win-rate dataset: `ncm` vs baselines

One row per LLM-judge judgment (pairs were judged in BOTH A/B orderings to debias position, so each (prompt, anchor-vs-baseline) pair appears twice).

—rows: 1800 (9 baselines x 100 prompts x 2 orderings)
—judge(s): gemini:gemini-2.5-pro
—baselines: bd3-l4, bd3-l8, gidd-base, gidd-small, mdlm-medium, mdlm-small, medium-gpt2, sedd, small-gpt2
—ncm_wins == "yes" count: 380 (21.1%); ties: 241

Schema

column	type	description
`sample_id`	int	0..99, identifies the shared OWT prompt
`baseline_model`	str	which baseline model `ncm` was compared against
`prompt`	str	the input prompt (identical for both continuations on a given row)
`ncm_generation`	str	`ncm`'s continuation
`baseline_generation`	str	the baseline's continuation
`judge_order`	str	`anchor_first` => NCM shown as "Response A"; `baseline_first` => NCM shown as "Response B"
`ncm_wins`	str	`"yes"` if the judge picked NCM, `"no"` otherwise (ties counted as "no")
`verdict`	str	raw verdict: `"win"` (NCM), `"loss"` (baseline), or `"tie"`

Judge prompt (the prompt sent to the LLM judge)

The judge received the following prompt for each pair. {PROMPT}, {RESPONSE_A}, {RESPONSE_B} are filled in from the dataset's prompt, and the two continuations in the order given by judge_order (so anchor_first => NCM is {RESPONSE_A}). The judge replies with strict JSON {"winner": "A" | "B" | "tie"}, which is what populates the verdict column.

You are comparing two continuations of the SAME prompt, each produced by a different small language model.

Decide which continuation is better overall, weighing:
- coherence: logical consistency, staying on-topic, and contextual appropriateness with respect to the prompt.
- fluency: natural, grammatical, readable English prose.

These are long-form generations from small models, so be slightly lenient and judge them relative to each other. Pick the single better continuation. Only answer "tie" if they are genuinely indistinguishable in overall quality.

Return strict JSON: {"winner": "A"} or {"winner": "B"} or {"winner": "tie"}.

PROMPT:
{PROMPT}

RESPONSE A:
{RESPONSE_A}

RESPONSE B:
{RESPONSE_B}

Notes for error analysis

—A pair contributes 2 rows (same sample_id + baseline_model, different judge_order). If the two rows disagree, the judge was position-sensitive on that pair.
—Generations are length-capped at the trained continuation length (128 BPE tokens).
—Prompts are identical across all baselines for a given sample_id (the win-rate harness verifies prompt alignment before pairing).