thefertilityplan/ivf-bench-orpo-qwen9b

IVF-Bench ORPO preference pairs

550 preference pairs for training a vision-language model to write clinical assessments of IVF embryo cases. Each row holds a day-5 blastocyst image, the case prompt, a preferred response, and a rejected one.

Built from IVF-Bench. The underlying embryo images and real clinical fields come from the Kromp et al. (2023) blastocyst dataset, released under CC BY 4.0.

How the pairs were made

Seven vision-language models answered each of the 550 test-split cases. A judge scored every answer on five rubrics from 1 to 5. For each case, the highest-scoring answer became chosen and the lowest-scoring became rejected.

Mean score gap between chosen and rejected is 1.49 points on the 1 to 5 scale.

Fields

The _full variants of the source JSONL additionally carry case_id, chosen_model, rejected_model, chosen_score, rejected_score, and score_gap for provenance.

Please read this before training on it

The chosen responses are close to a monoculture. One model produced 92% of them (460 of 500 in the train split). Anything trained on this data inherits that model's style, its hedging patterns, and its blind spots. It also means a judge from the same family will tend to score the result generously. We discuss this at length in the paper and it is the single largest caveat on the dataset.

Rejected responses skew toward two models, mostly a small open model and a fast commercial one, so the pairs teach "avoid these failure modes" more than they teach "prefer this reasoning."

The pairs come from the benchmark's test split. A model trained on them is in-domain on that split. Evaluate on the 103-case held-out split instead.

Thirty of the 550 pairs were built from six responses rather than seven, because the base Qwen 3.5-9B did not complete those cases.

Part of the patient context is generated. Age, AMH, endometrial thickness, oocyte counts, transfer day, and outcomes are real. BMI, FSH, diagnosis, protocol, cycle history, partner age, semen parameters, and lifestyle are sampled from published population distributions. Each case names its generated fields.

Not for clinical use. Research artifact only.

Citation

@article{correa2026ivfbench,
  author  = {Correa, Andrew G. A. and Yoon, Brittany},
  title   = {{IVF-Bench}: A Rubric-Based Standard for Evaluating
             Vision-Language Models on {IVF} Clinical Reasoning},
  journal = {arXiv preprint},
  year    = {2026}
}

Please also cite the source dataset:

@article{kromp2023,
  author  = {Kromp, Florian and others},
  title   = {An annotated human blastocyst dataset to benchmark deep learning
             architectures for in vitro fertilization},
  journal = {Scientific Data},
  volume  = {10},
  pages   = {271},
  year    = {2023},
  doi     = {10.1038/s41597-023-02182-3}
}