chongpangnasilemak/medical-terminology-mcq

Medical Terminology MCQ

742 multiple-choice items on medical terminology — word-building (root / prefix / suffix), abbreviation expansion and disambiguation, minimal-pair discrimination, and compositional generalization to novel compounds.

Labels are what GPT-5.6-sol ruled they are. Items were written by Claude and adjudicated by GPT-5.6-sol; where the two disagreed, the adjudicator's ruling settled the item. That makes GPT-5.6-sol the source of truth for every answer key here. No clinician or medical educator reviewed any item at any stage. There has been no medical validation of any kind. Two language models agreeing is evidence of consistency, not of correctness — a misreading they share cannot surface as disagreement. Do not use this for anything clinical.

Why this exists

The objective was to evaluate smaller models on ICD coding capability, which needed a proxy specific to the ICD context rather than a general medical exam. Coding from documentation has a vocabulary prerequisite: a model that cannot expand an abbreviation, or that reads -ostomy and -otomy as the same operation, misreads the record before any coding rule applies. This dataset measures that prerequisite on its own, so a failure at the coding layer can be attributed rather than guessed at.

Two companion datasets cover the coding layer above it:

• —ICD-10-CM Coding MCQ — diagnosis coding conventions.
• —ICD-10-PCS Coding MCQ — inpatient procedure coding.

Usage

from datasets import load_dataset

ds = load_dataset("chongpangnasilemak/medical-terminology-mcq", split="test")
print(ds[0]["question"], ds[0]["choices"], ds[0]["answer"])

MMLU / lm-evaluation-harness row shape, so it runs under existing MCQ harnesses unmodified.

Structure

Splits. test = 673 items (the evaluation set). dev = 69 items, disjoint, intended as few-shot exemplars — capped at a fifth of each category so rare categories are not drained out of test.

Answer positions in test are near-uniform (169/168/168/168), so a constant-answer baseline scores 25.0%. Difficulty: 198 easy / 303 medium / 172 hard.

Categories

Two subsets are worth reporting separately. The 76 novel_compound items are contamination-resistant: morphologically valid terms that appear neither in the source curriculum nor in standard usage, so they cannot be recalled, only composed. And abbrev_context contains 20 letter-strings that appear more than once with different correct answers (PE as both physical examination and pulmonary embolism, MS in three senses); every option is a genuine expansion, so only the clinical context resolves them.

Reference results

Chance is 25.0%. unparsed counts replies that never committed to a letter; they are scored wrong. Zero-shot, single-letter answer, default reasoning effort, 4,000-token budget.

Only the GPT-5.x family is reported, because it answered with a bare letter on every call across all three datasets — zero unparsed — so these numbers reflect the items rather than any model's output formatting. GPT-5.6-sol is excluded: it defined the labels, so scoring it would measure its agreement with itself.

This dataset saturates above roughly frontier-mini scale. Read it as a floor test for whether a model has the vocabulary at all, not as a ranking instrument.

Provenance and limitations

Please read this before using the dataset for anything consequential.

• —Items are authored by Claude (Anthropic) and adjudicated by GPT-5.6-sol. There has been no expert human annotation and no clinical review. If you need clinician-verified gold, this is not that dataset.
• —The audit was two-pass: GPT-5.6-sol answered all 673 test items blind (stem and options only, never the key), agreeing with our key on 673/673, with no disagreements to adjudicate.
• —Validation is structural and consistency-based — schema conformance, distractor quality rules, duplicate detection, position balance — not medical adjudication.
• —The source is an introductory curriculum aimed at pre-med students and non-clinical professionals. It is a vocabulary evaluation, not a test of clinical competence.
• —Scope is deliberately narrow: roughly 100 morphemes, 50 diseases, 50 procedures, a 64-drug table and a handful of eponyms.

Source and credit

Built from the DMU Online Medical Terminology Course, a free public curriculum from Des Moines University:

<https://www.dmu.edu/medterms/>

Chapter pages used include Basics, Musculoskeletal System, Circulatory System, Digestive System and Cancer Terms. SOURCES.md lists all 43 pages with their URLs.

All credit for the underlying curriculum belongs to Des Moines University. This dataset does not redistribute the course: it contains only derived assessment items — stems, options and rationales written independently, with terminology facts used as the facts they are.

Citation

@misc{medical_terminology_mcq,
  title  = {Medical Terminology MCQ: an evaluation set derived from the
            DMU Online Medical Terminology Course},
  note   = {Items authored by Claude (Anthropic), adjudicated by GPT-5.6-sol;
            not expert-reviewed},
  year   = {2026},
  url    = {https://huggingface.co/datasets/chongpangnasilemak/medical-terminology-mcq}
}