chongpangnasilemak/icd10cm-coding-mcq

ICD-10-CM Coding MCQ

403 multiple-choice items on ICD-10-CM diagnosis coding — whether a model can apply the classification's conventions to documentation it is handed: Excludes1 and Excludes2 notes, 7th-character selection, placeholder X, laterality, Index→Tabular verification, combination codes, specificity.

Labels are what GPT-5.6-sol ruled they are. Items were written by Claude and adjudicated by GPT-5.6-sol; where the two disagreed, the adjudicator's ruling settled the item. That makes GPT-5.6-sol the source of truth for every answer key here. No certified medical coder or clinician reviewed any item at any stage. There has been no professional coding validation or clinical review. Two language models agreeing is evidence of consistency, not of correctness — a misreading they share cannot surface as disagreement. Nothing here is coding guidance, and items must not be used to justify a code assignment on a real claim.

Why this exists

The objective was to evaluate smaller models on ICD-10-CM coding capability. General medical evaluations do not answer that question — they test clinical knowledge, not whether a model applies classification conventions to documentation. We needed a proxy specific to the ICD context, cheap enough to run repeatedly across a size ladder.

Two companion datasets sit either side of it:

• —Medical Terminology MCQ — the vocabulary prerequisite underneath, so a model that fails here can be checked against whether it understood the terminology in the first place.
• —ICD-10-PCS Coding MCQ — inpatient procedure coding. A CM code is looked up; a PCS code is built from seven independent character axes, and the two capabilities come apart.

It does not test memorization of the code set. Every item is self-contained: whatever Index entry, Tabular note, code string or documentation excerpt the question turns on is supplied in the stem. A model that has memorized all of ICD-10-CM gains nothing — it has to know what the conventions oblige it to do with material it is shown. The design test each item had to pass: could a competent coder answer this with the book open in front of them?

Usage

from datasets import load_dataset

ds = load_dataset("chongpangnasilemak/icd10cm-coding-mcq", split="test")
print(ds[0]["question"], ds[0]["choices"], ds[0]["answer"])

MMLU / lm-evaluation-harness row shape, so it runs under existing MCQ harnesses unmodified.

Structure

Splits. test = 350 items (the evaluation set). dev = 53 items, disjoint, intended as few-shot exemplars.

Answer positions in test are near-uniform (88/88/87/87), so a constant-answer baseline scores 25.0%. Difficulty: 111 easy / 153 medium / 86 hard.

Categories

`excludes1` is deliberately two-directional. Of its 27 items, 10 cover the Official Guidelines exception — two unrelated conditions may both be reported despite an Excludes1 note, and where relatedness is unclear the provider is queried. Without those, a model scores full marks by learning "Excludes1 means never both", which is the wrong rule. The same balancing applies to signs_symptoms and specificity.

Known limitation in `seventh_character`. 11 test items have options whose text is a letter (A, D, S, G, K, P), so the answer space overlaps the A–D option labels. A model replying "S" may mean the 7th character rather than option S. Resolve such replies against the option text, or exclude the category if that matters to you.

Reference results

Chance is 25.0%. unparsed counts replies that never committed to a letter; they are scored wrong. Zero-shot, single-letter answer, default reasoning effort, 4,000-token budget.

Only the GPT-5.x family is reported, because it answered with a bare letter on every call across all three datasets — zero unparsed — so these numbers reflect the items rather than any model's output formatting. GPT-5.6-sol is excluded: it defined the labels, so scoring it would measure its agreement with itself.

The 27-point spread between the smallest and largest model shown is the point of the dataset: unlike its terminology companion, this one still discriminates well below frontier scale.

Provenance and limitations

Please read this before using the dataset for anything consequential.

• —Items are authored by Claude (Anthropic) and adjudicated by GPT-5.6-sol. There has been no expert human annotation and no professional coding review. If you need coder-verified gold, this is not that dataset.
• —The audit was two-pass: GPT-5.6-sol answered all 350 test items blind (stem and options only, never the key), agreeing with our key on 349/350, including all 86 hard items. The single disagreement was adjudicated in favour of the existing key.
• —Validation is structural and consistency-based — schema conformance, distractor quality, duplicate detection, position balance — not coding adjudication.
• —The source is a 23-minute awareness-level course, not a coding certification syllabus. It is a conventions evaluation, not a test of professional coding competence.
• —Codes and Index entries were cross-checked against a local ICD-10-CM code set where possible, but code sets change annually and items are not pinned to a specific fiscal year.
• —Not for clinical or billing use.

Source and credit

Built from "Diagnosis Coding: Using the ICD-10-CM", a free web-based training from the Centers for Medicare & Medicaid Services Medicare Learning Network (MLN):

<https://www.cms.gov/Outreach-and-Education/MLN/WBT/MLN6447308-ICD-10-CM/icd10cm/story.html>

The extracted lesson text is included, under source/cms/. CMS materials are works of the U.S. Federal Government and are in the public domain, so redistributing them is permitted and makes every item auditable against what it was written from. Coding conventions were additionally checked against the ICD-10-CM Official Guidelines for Coding and Reporting.

All credit for the underlying curriculum belongs to CMS. The assessment items — stems, options and rationales — were written independently and are not part of the course.

Citation

@misc{icd10cm_coding_mcq,
  title  = {ICD-10-CM Coding MCQ: a coding-conventions evaluation set derived from the
            CMS MLN course "Diagnosis Coding: Using the ICD-10-CM"},
  note   = {Items authored by Claude (Anthropic), adjudicated by GPT-5.6-sol;
            not expert-reviewed},
  year   = {2026},
  url    = {https://huggingface.co/datasets/chongpangnasilemak/icd10cm-coding-mcq}
}