Datasetpublic

jason-brelsford/hla-bench

HLA-Bench — contamination-resistant evaluation of LLMs on clinical immunogenetics Every model family tested — Claude, Qwen, Mistral, Llama, Phi, Gemma — scores 0% on two-field ambiguity expansion, the core clinical trap in HLA typing (a 2-field name like A*02:01 denotes 2–389 full-resolution alleles): 0 of 30 tasks for every model. Models fabricate allele names at 0.06–0.20 per task across the nine models run on the full 550-task suite. The claude-sonnet-4-6 rate of 0.09 is a… See the full description on the dataset page: https://huggingface.co/datasets/jason-brelsford/hla-bench.

sourceHugging Faceapache-2.0updated 6d agoView on Hugging Face

0likes150downloads

Dataset Card

HLA-Bench — contamination-resistant evaluation of LLMs on clinical immunogenetics

Every model family tested — Claude, Qwen, Mistral, Llama, Phi, Gemma — scores 0% on two-field ambiguity expansion, the core clinical trap in HLA typing (a 2-field name like A*02:01 denotes 2–389 full-resolution alleles): 0 of 30 tasks for every model. Models fabricate allele names at 0.06–0.20 per task across the nine models run on the full 550-task suite. The claude-sonnet-4-6 rate of 0.09 is a lower bound, because 187 of its 550 responses were truncated and graded malformed. On donor–recipient matching, naive string manipulation collapses to 0% and open models reach 0–14% — they count matched loci instead of chromosomes.

HLA-Bench grades every answer with an executable oracle computed from the pinned IPD-IMGT/HLA release itself — no human labels, no LLM judges. Ground truth regenerates deterministically from each quarterly database release, so a versioned share of tasks is post-training-cutoff by construction.

What's in this dataset

The public dev splits only:

—family_a_dev/ — 112 nomenclature tasks (of 550; truncation, G/P groups, serology, rename history, ambiguity, near-miss traps)
—family_c_dev/ — 43 donor–recipient matching pairs (of 205; 6/6–12/12 frameworks, null-allele traps, GvH/HvG directionality)

The scored splits are sealed and regenerate per release from (tag, seed) — that's the contamination-resistance design. Run them yourself from the source:

bash

git clone https://github.com/jasonbrelsford/verifiable-science-envs
pip install -e ".[dev]"
hla-bench generate && hla-bench run ollama/qwen2.5:7b --suite runs/hla-bench-a --split dev

Links

—Code, graders, full results with CIs: https://github.com/jasonbrelsford/verifiable-science-envs
—In-browser verifier demo (nothing leaves your machine): https://hlaverify.com/demo
—For AI agents: https://hlaverify.com/llms.txt
—Evaluation/training licensing & partnerships: hello@hlaverify.com (Brelsford Software LLC)

Tasks are Apache-2.0. Reference data are not included: graders fetch IPD-IMGT/HLA at runtime (CC-BY-ND, Barker DJ et al., NAR 2025) and never redistribute it. This card is maintained by Claude on behalf of Jason Brelsford.