hanseungwook/medmcqa-filtered-v16

MedMCQA Filtered v16

This commit adds normalized source explanations to official validation for a mixed gold-rationale-assisted likelihood evaluation. The immutable training revision remains 29cf55f62c523dc9c1c45e2d77f52208058e7d19; evaluation clients must pin the full SHA of this commit separately.

Splits

Validation has 2,204 rows with rationales and 1,977 without rationales. Gold-label counts are A=1,348, B=1,084, C=924, and D=825.

Two rows exceeded the 1,024-token teacher layout and were permanently removed: 2e78ce60-b4af-41c4-b2bd-595a6e728c1d (1,043 tokens) and 788eb150-1cd1-4895-b929-56ffa68eae61 (1,062 tokens). Full source indices, labels, subjects, lengths, and provenance are in artifacts/validation_exclusions.json and artifacts/manifest.json.

Exact context layout

Lengths use separately tokenized segments matching teacher training:

1. 1.formatted question with its terminal newline;
2. 2.normalized rationale with a terminal newline when present;
3. 3.The answer is: X;
4. 4.one terminal GPT-2 EOS.

Validation rationales are normalized using the same HTML unescape, Unicode NFKC, newline, and whitespace rules used for training. They are not truncated, edited, or filtered for answer declarations. Existing structural leakage-screen results are annotations only. Semantic overlap between train and validation continues to use all 4,183 original validation rows (4,169 unique semantic groups) before the two length exclusions are applied.

For generation evaluation, use only question; the stored cot is not an input. Likelihood results that use cot are mixed gold-rationale-assisted measurements, not ordinary closed-book accuracy.

Reproduction and limitations

Pinned builder sources are under builder/; the complete release contract and split fingerprints are in artifacts/manifest.json. Explanations originate from the upstream dataset and may reveal answers, contain mistakes or outdated medical claims, and must not be used for clinical decisions.