rohan2810/medmcqa50

MedMCQA-50

This dataset expands `rohan2810/medmcqa` from 20 to 50 candidates per example for finite-pool preference-optimization experiments.

Construction

For every example, all unique candidates in the original 20-entry pool are preserved. Duplicate answer strings in the source are collapsed while retaining their first occurrence. Additional distractors are sampled deterministically from the 172,635-answer source candidate universe using seed 1958 until each row has exactly 50 unique candidates. The resulting candidates are deterministically shuffled, and the candidate list embedded in fixed_prompt is replaced accordingly.

No model scores, embeddings, selector objectives, or evaluation outcomes are used to construct the candidate pools. This nested construction limits dataset-generation confounding when studying a larger candidate pool.

Splits

• —train: 98,864 rows
• —validation: 12,358 rows
• —test: 12,359 rows

Each row contains:

• —fixed_prompt: the medical question and the same 50 answer candidates.
• —itemList: 50 unique answer candidates.
• —trueSelection: the true answer, appearing exactly once in itemList.

Provenance

• —Source repository: rohan2810/medmcqa
• —Source revision: b4944b5c7085a408c24f50f4b2ae8eed0ef66239
• —Output repository: rohan2810/medmcqa50
• —Generator seed: 1958
• —Generator: neurips_rebuttal/medmcqa50_dataset/build_medmcqa50.py