chongpangnasilemak/icd10cm-exam-mcq

ICD-10-CM Exam MCQ

203 multiple-choice items drawn from ICD-10-CM practice exams and coursework — clinical vignettes requiring an actual code assignment, plus questions on conventions, guideline structure and Chapter 20 external-cause rules.

## ⚠️ Read this before using or redistributing The questions are third-party material of unverified provenance, reproduced verbatim. They come from practice exams and coursework — one identifies itself as belonging to an "ICD-10-CM Preparation Class" — obtained as text files whose original names were numeric document ids. The copyright holder is unknown and no licence was granted. Nothing here is a derived work: the questions are someone else's. If you are the rights holder and object, the dataset will be removed. No certified medical coder or clinician reviewed any item or any key.

Two kinds of answer key, and the difference matters

This is the one thing to understand before using the dataset. The four sources did not arrive in the same condition, and roughly half the items carry a key that was reconstructed rather than recovered.

The derived keys. Both practice exams stated that "the answers are highlighted" — and the highlighting was visual formatting that did not survive text extraction. Each item was answered independently by gpt-5.6-sol and gpt-5.6-terra, with the official FY2023 description of every ICD-10-CM code appearing in any option supplied in the prompt (496 of 499 codes resolved), so the models applied rules to verified facts rather than recalling what T45.1X5A means. They agreed on 127 of 132 (96.2%). The 5 disagreements were dropped, not adjudicated — they were genuinely defective, one containing a typo in an option (Y92,830, comma for decimal point). Those keys have never been checked against the original, and two-model consensus is a weaker warrant than a published key.

The published keys. The worksheet arrived with its answer key intact. The Chapter 20 exam marked its correct option by highlighting too, but there the highlight failed differently: the correct option lost its A/B/C/D label rather than its emphasis, so it is identifiable, and each question additionally carried an "Expected answer" note citing the guideline it rests on. Both signals agree on all 20.

One measured error in the derived keys. Where the worksheet and practice exam B ask the same question, the published key contradicts the derived one. For a patient with insomnia and upset stomach whose physician diagnoses impending menopause, the derived key chose N95.9; the published key gives G47.00, K30. The published key is right — ICD-10-CM's impending or threatened condition guideline says to code the presenting signs and symptoms when the condition did not occur. It is a single overlapping item and not a rate, but it is a concrete instance of the failure mode the derived keys are exposed to.

How this differs from ICD-10-CM Coding MCQ

Its sibling, icd10cm-coding-mcq, is built on one rule: every item supplies the Index entry, Tabular note or code string it turns on — "could a competent coder answer this with the book open?" It deliberately does not reward memorising the code set.

This dataset is the opposite. Nothing is supplied:

"Patient with thyroid cancer has fever and found to have chemo drug induced agranulocytosis." a. D72.0, T36.91XA, C73, R50.81 b. D70.1, T45.1X1A, C73 c. … d. D70.1, T45.1X5A, C73, R50.81

You either know the code set or you do not. That makes the pair useful together: the gap between a model's two scores separates applying conventions from recalling codes.

Structure

MMLU / lm-evaluation-harness row shape. test = 193 items, dev = 10, disjoint.

Option order was reshuffled so answer positions are uniform (49/48/48/48) and a constant-answer baseline scores 25.0%. The sources were skewed toward later positions.

The sibling datasets carry chapter, difficulty, rationale, source_url and is_novel_compound. Those are omitted here because they would say nothing: two of them are constant on every row, rationale is near-identical boilerplate, and a difficulty label derived by counting codes in an option is not the considered judgment the other datasets use. What little they encode is described on this card instead.

Reproducing a run

Zero-shot, single-letter answer, thinking enabled where supported, sampling per each model's own card — never greedy, because at temperature=0 small reasoning models fall into verbatim repetition loops and generate to the token ceiling without answering.

A lm-evaluation-harness task config ships alongside at lm_eval_tasks/, scoring by log-probability over the four options — deterministic, no parsing — though it cannot be used with a model's thinking mode or against hosted reasoning APIs that do not expose logprobs.

Limitations

• —Not for clinical or billing use. Nothing here is coding guidance.
• —Roughly half the keys are model-derived and unverified against the source. See above.
• —Codes are fiscal-year specific. Keys were checked against FY2023; three codes appearing in options (J72.0, Y91.019, Y91.830) are not in that set at all.
• —Small. 193 test items; a 5-point difference is inside the noise.
• —No human validation of items or keys, at any stage.