Wilhelm-Foundation/rare-archive-eval-rarearena-rdc

RareArena RDC — Rare Disease Cases Evaluation Benchmark

4,376 clinical vignettes with laboratory test results across rare diseases for evaluating AI diagnostic reasoning with lab data. Part of the Rare AI Archive.

Research use only. This dataset is an evaluation benchmark for AI systems. It is NOT intended for clinical decision-making and should NOT be used as a diagnostic tool.

Dataset Description

• —Repository: Wilhelm-Foundation/rare-archive-eval-rarearena-rdc
• —License: CC BY-NC-SA 4.0
• —Version: 0.1.0
• —Part of: Rare AI Archive · Complete Toolkit Collection

How RDC Differs from RDS

RDC cases are generally more complex — they require the model to integrate laboratory findings with clinical presentation, closer to real-world diagnostic workflows.

Ecosystem Context

RDC cases add a critical dimension to evaluation: laboratory test results. In the full agentic diagnostic system, models learn to invoke clinical tools like ClinVar and gnomAD, then interpret their results alongside clinical presentations. This dataset tests that capability — can the model integrate lab data into its diagnostic reasoning?

This maps directly to Stage 2 of the 4-stage training pipeline, where models learn to invoke tools and interpret real API responses. RDC evaluation measures whether tool-augmented reasoning produces better differentials than clinical presentation alone.

Disease categories in this dataset map to the ontology's clustering scheme, enabling evaluation of condition-specific model adapters alongside the foundation model.

Dataset Structure

Data Fields

Data Splits

Message Format

Each record follows the OpenAI chat format:

1. 1.System: Expert rare disease diagnostician instruction
2. 2.User: Clinical vignette with laboratory/diagnostic test results
3. 3.Assistant: Expected diagnostic reasoning and differential

Dataset Creation

Source Data

Derived from the RareArena RDC (Rare Disease Cases) benchmark. Cases are drawn from published case reports in medical literature, rewritten by GPT-4o. Test results are concatenated with clinical vignettes to form comprehensive presentations.

Data Processing

1. 1.Case reports sourced from published medical literature
2. 2.Clinical vignettes + test results generated via GPT-4o (de-identification)
3. 3.Test results concatenated to clinical vignettes in the user message
4. 4.Formatted as OpenAI chat JSONL via rare-archive-datasets v0.1.0 parse_case() (v3 format)

PHI Status

no_phi — All vignettes are GPT-4o rewrites of published case reports. No real patient data.

Intended Use

Primary use: Evaluation benchmark for rare disease diagnostic AI models, specifically testing the ability to integrate laboratory findings with clinical reasoning.

• —Measure Top-K differential diagnosis accuracy with lab context
• —Compare model performance: RDS-only vs RDC (does lab data improve accuracy?)
• —Benchmark lab result interpretation in diagnostic reasoning

NOT intended for: Clinical decision-making, patient diagnosis, or model training.

Part of the ecosystem flywheel: RDC evaluation reveals how well models interpret clinical data — a key input for Arena evaluators who score Tool Usage as one of the 5 quality dimensions.

Bias, Risks & Limitations

Known Biases

Risks

• —Not for clinical use: Evaluation benchmark only
• —Lab interpretation limits: AI performance on curated lab results does not reflect performance on real clinical lab reports
• —Smaller sample: 4,376 cases provides less statistical power than RDS for rare disease subgroup analysis

Limitations

• —Monolingual (English only)
• —Smaller than RDS (4,376 vs 8,562) — fewer diseases covered
• —Lab results are text descriptions, not structured lab values
• —No temporal ordering of tests within a case

Loading the Dataset

Using HuggingFace Datasets

from datasets import load_dataset

# Load the full dataset
dataset = load_dataset("Wilhelm-Foundation/rare-archive-eval-rarearena-rdc")

# Access the test split
test = dataset["test"]
print(f"Total cases: {len(test)}")

# Inspect a case with lab results
example = test[0]
for msg in example["messages"]:
    print(f"[{msg['role']}] {msg['content'][:150]}...")

Using Pandas

import pandas as pd

df = pd.read_parquet(
    "hf://datasets/Wilhelm-Foundation/rare-archive-eval-rarearena-rdc/data/test-00000-of-00001.parquet"
)
print(f"Shape: {df.shape}")

Comparing RDS vs RDC Performance

from datasets import load_dataset

rds = load_dataset("Wilhelm-Foundation/rare-archive-eval-rarearena-rds", split="test")
rdc = load_dataset("Wilhelm-Foundation/rare-archive-eval-rarearena-rdc", split="test")

print(f"RDS cases: {len(rds)} (clinical vignettes only)")
print(f"RDC cases: {len(rdc)} (vignettes + lab results)")

# Find overlapping disease IDs
rds_diseases = set(x["metadata"]["disease_id"] for x in rds)
rdc_diseases = set(x["metadata"]["disease_id"] for x in rdc)
overlap = rds_diseases & rdc_diseases
print(f"Shared diseases: {len(overlap)}")

Related Resources

Citation

@misc{rarearena2024,
  title={RareArena: A Benchmark for Rare Disease Diagnosis},
  author={Zhao, Zhiyu and others},
  year={2024},
  url={https://github.com/zhao-zy15/RareArena}
}

A program of the [Wilhelm Foundation](https://wilhelm.foundation). Built on [Lattice Protocol](https://github.com/LatticeProtocol). No disease is too rare to matter.