Parexel/clinical-trials-qa

Clinical Trials QA Dataset

A multi-tier question-answering benchmark for evaluating Retrieval-Augmented Generation (RAG) systems on clinical trial protocols from ClinicalTrials.gov.

Dataset Summary

This dataset provides question-answer pairs across four difficulty tiers, designed to benchmark RAG systems on real-world clinical trial documentation. Questions span four reasoning categories and require retrieval from protocol PDFs.

Key Features:

• —4 difficulty tiers based on corpus size (13 to ~45,000 protocols)
• —18 question templates across 4 reasoning types
• —Ground truth answers with SQL evidence
• —Closed-set and open-ended questions
• —Natural language phrasing with proper formatting

Dataset Structure

Data Instances

{
  "id": "a3f2e9b8c1d4...",
  "question": "What is the most common primary purpose among trials investigating Behavioral interventions?",
  "answer": "Treatment",
  "corpus_size": 129,
  "question_type": "Evidence Synthesis",
  "possible_answers": ["Treatment", "Prevention", "Diagnostic Test", "Health Services Research"],
  "qa_template": "synthesis_ranking",
  "sql_query": "SELECT primary_purpose, COUNT(*) as frequency FROM trials WHERE intervention_type = 'BEHAVIORAL' GROUP BY primary_purpose ORDER BY frequency DESC",
  "sql_response": [
    {"primary_purpose": "TREATMENT", "frequency": 42},
    {"primary_purpose": "PREVENTION", "frequency": 15}
  ],
  "evidence_size": 5
}

Data Fields

Core Fields:

• —id (string): SHA256 hash-based unique identifier
• —question (string): Natural language question
• —answer (string): Ground truth answer
• —corpus_size (int): Number of protocols in corpus (13/129/1291)

Evaluation Fields:

• —possible_answers (list[string]): Valid answer options (null for open-ended)
• —sql_response (list[dict]): SQL query results supporting the answer
• —evidence_size (int): Number of evidence records needed

Metadata Fields:

• —question_type (string): Question category
• —Cross-Comparison
• —Evidence Synthesis
• —Multi-Hop Reasoning
• —Subset Reasoning
• —qa_template (string): Template name used for generation
• —sql_query (string): SQL query for reproducibility

Data Splits

Usage

Load Dataset

from datasets import load_dataset

# Load specific tier
tier_b = load_dataset("Parexel/clinical-trials-qa", split="tier_b")

# Load all tiers
dataset = load_dataset("Parexel/clinical-trials-qa")

# Access fields
for item in tier_b:
    print(f"Q: {item['question']}")
    print(f"A: {item['answer']}")
    print(f"Corpus: {item['corpus_size']} protocols")

Filter by Question Type

# Get only evidence synthesis questions
synthesis = tier_b.filter(
    lambda x: x["question_type"] == "Evidence Synthesis"
)

# Get closed-set questions (have possible_answers)
closed_set = tier_b.filter(
    lambda x: x["possible_answers"] is not None
)

Data Collection

Question Generation

Questions were automatically generated using 18 templates across 4 reasoning categories:

Cross-Comparison (6 templates):

• —Comparing properties across trials
• —Percentage calculations
• —Prevalence analysis
• —Temporal comparisons

Evidence Synthesis (6 templates):

• —Ranking most/least common values
• —Frequency analysis
• —Multi-filter aggregation
• —Single-filter queries

Multi-Hop Reasoning (3 templates):

• —Combining date and enum filters
• —Sequential property lookups
• —Complex conditional logic

Subset Reasoning (3 templates):

• —Set intersections
• —Set differences
• —Common elements

Quality Controls

• —Null filtering: Removed NA/NULL/UNKNOWN values
• —Natural formatting: Enum values converted to readable form (e.g., "PHASE3" → "Phase 3")
• —Selectivity filtering: Ensured questions have meaningful answers
• —Duplicate detection: Removed redundant questions
• —Answer validation: Verified answers match evidence

Protocol Sampling

• —Tier A: Random sample of 13 protocols (seed=42)
• —Tier B: Stratified sample of 129 protocols across 9 strata (seed=42)
• —Tier C: Stratified sample of 1,291 protocols across 9 strata (seed=42)
• —Tier D: All remaining protocols (~45,000), excluding Tier A/B/C to avoid duplicates

Stratification based on:

• —Study type (Interventional/Observational)
• —Phase (Early/Late/NA)
• —Study status (Recruiting/Completed/Other)

Intended Use

Primary Use Case

Benchmarking Retrieval-Augmented Generation (RAG) systems on clinical trial data.

Evaluation Pipeline:

1. 1.Load protocol PDFs into your RAG system
2. 2.Query with questions from dataset
3. 3.Compare system answers to ground truth
4. 4.Calculate accuracy/F1 scores

Evaluation Metrics

Closed-set questions (have possible_answers):

• —Exact match accuracy
• —Top-k accuracy

Open-ended questions:

• —F1 score (token overlap)
• —ROUGE scores
• —Semantic similarity

Evidence retrieval:

• —Compare retrieved protocols to nct_ids (available in full schema)
• —Precision/recall of evidence retrieval

Limitations

1. 1.Synthetic Generation: Questions are template-generated, not human-authored
2. 2.English Only: All content in English
3. 3.ClinicalTrials.gov Only: Limited to US-registered trials
4. 4.Structured Data: Questions derived from structured fields, not free-text
5. 5.Protocol PDFs: Requires access to protocol PDFs (not included in this dataset)

Accessing Protocol PDFs

Protocol PDFs are available in a separate repository (coming soon):

from huggingface_hub import hf_hub_download

# Download specific protocol
pdf_path = hf_hub_download(
    repo_id="your-org/clinical-trials-protocols",
    filename="tier_b/NCT02059408.pdf",
    repo_type="dataset"
)

Alternatively, download directly from ClinicalTrials.gov using the NCT IDs.

Citation

If you use this dataset in your research, please cite:

@dataset{clinical_trials_qa_2026,
  title={Clinical Trials QA: A Multi-Tier RAG Benchmark},
  author={Parexel},
  year={2026},
  publisher={Hugging Face},
  url={https://huggingface.co/datasets/Parexel/clinical-trials-qa}
}

License

• —QA Pairs: CC BY 4.0 - Free to use with attribution
• —Protocol PDFs: Public domain (ClinicalTrials.gov terms)

Dataset Creators

Created by Parexel.

Generated using automated question templates with quality controls for selectivity, natural phrasing, and answer validation.

Additional Information

Dataset Version

• —Version: 1.0.0
• —Release Date: 2026
• —Schema Version: 1.0

Contact

For questions or issues, please open an issue on the dataset repository.

Acknowledgments

• —ClinicalTrials.gov for providing public access to clinical trial data
• —Hugging Face for dataset hosting infrastructure