Clinical-Reasoning-Hub/pentabrid-reproducibility
Pentabrid 27B: reproducibility package Everything required to recompute the results of a controlled evaluation of fine-tuning configurations for medical question answering. Openly available with no access restrictions. Contents Path Description per_item/medxpertqa_*.jsonl Per-item predictions for all six checkpoints on 2,450 MedXpertQA-Text items. Fields: id, gold, extracted_answer, correct, explicit_marker_present, n_markers, response_chars… See the full description on the dataset page: https://huggingface.co/datasets/Clinical-Reasoning-Hub/pentabrid-reproducibility.
Pentabrid 27B: reproducibility package
Everything required to recompute the results of a controlled evaluation of fine-tuning configurations for medical question answering. Openly available with no access restrictions.
Contents
The scorer used for every reported accuracy is analysis/merge_medxpertqa.py. An identical copy is retained at scripts/merge_medxpertqa.py.
Recomputing the reported results
All accuracy, retention, acquisition and turnover figures can be recomputed directly from per_item/. Model weights are not required. Retention is the proportion of parent-correct items a derivative also answered correctly; acquisition is the proportion of parent-incorrect items it answered correctly; turnover is the proportion of all items whose binary correctness differed from the parent.
What the study found
At a fixed parent checkpoint, accuracy spanned 16.86 percentage points across configurations. No derivative exceeded the parent. Derivatives within 2.41 points of the parent nevertheless changed 15.1% to 18.2% of individual item outcomes, so aggregate accuracy is a poor summary of behavioural change. The pattern reproduces on MedQA and MedMCQA.
Three analyses returned null results and are reported as such: generated rationale length did not explain the accuracy differences; turnover was not concentrated in any medical task, body system or question type after correction for multiplicity; and imposed reasoning protocols did not improve accuracy at inference.
Scope and limitations
Configurations differ in several training variables simultaneously, so the design does not isolate any one of them. One training run was performed per configuration. The analysis is exploratory, since the primary benchmark partly informed the design of later configurations. Benchmark question text is not redistributed; MedXpertQA, MedQA and MedMCQA are available from their original sources under their own licences. The derived training corpus is not redistributed because it draws on licensed examination content, but the scripts that generate and decontaminate it are included here.
Related
Model checkpoints: <https://huggingface.co/Clinical-Reasoning-Hub/pentabrid-27b> (DOI 10.57967/hf/9500)
Dataset DOI: 10.57967/hf/9501
Authors and licence
Dr. Adnan Agha (ORCID 0000-0002-2704-8931) and Dr Eram Anwar (ORCID 0009-0006-9335-9208), College of Medicine and Health Sciences, United Arab Emirates University. CC BY 4.0.
Updated September 2026 to align with the peer-reviewed analysis. An earlier version of this card stated that the specialisation procedure determines medical reasoning performance and referred to a commentary submitted elsewhere. That causal interpretation was not supported by item-level analysis and has been withdrawn.