MJ16/pharmacoeconomic-evidence-extraction-dataset

Pharmacoeconomic Evidence Extraction Dataset

License

This dataset is released under the Creative Commons Attribution 4.0 International (CC BY 4.0) license. You may use, share, and adapt the dataset provided that appropriate credit is given to the dataset authors.

For the full license terms, see the CC BY 4.0 license.

Overview

This dataset contains 250 expert-annotated records for research on automated extraction of structured pharmacoeconomic and clinical-trial evidence from published clinical trial reports.

The dataset was developed as part of the research study:

"Credibility-Aware Pharmacoeconomic Evidence Extraction from Clinical Trials via RAG-Enhanced Fine-Tuned Small Language Models"

Each record contains the publication title, abstract, and expert-produced structured annotations covering trial metadata, population characteristics, intervention/comparator arms, and reported outcomes.

Dataset Structure

The dataset is distributed in JSON Lines (`.jsonl`) format, with one JSON object per record. Each record follows a fixed structured format.

Record Format

{
  "trial_id": "PMID_XXXXXXXX",
  "title": "<clinical trial publication title>",
  "abstract": "<abstract text>",
  "labels": {
    "trial_metadata": {
      "registry_source": "<registry source>",
      "trial_phase": "<trial phase>",
      "study_design": "<study design>",
      "blinding": "<blinding status>",
      "years_of_enrollment": "<enrollment period>"
    },
    "population": {
      "disease": "<disease>",
      "total_number_of_patients": "<number of patients>"
    },
    "arms": [
      {
        "arm_name": "<arm name>",
        "arm_type": "<intervention/comparator/not_specified>"
      }
    ],
    "outcomes": [
      {
        "abbreviation": "<outcome abbreviation>",
        "expansion": "<outcome full name>"
      }
    ]
  }
}

The top-level trial_id identifies the source publication, while the remaining trial-level information is stored under labels.trial_metadata.

Annotation Schema

The annotation schema is grounded in the PICO framework (Population, Intervention, Comparator, Outcome), a standard framework used in clinical research and evidence synthesis. It is adapted here into a fixed, schema-constrained structure designed for structured pharmacoeconomic evidence extraction.

The schema was defined in collaboration with a pharmacoeconomic domain expert and implemented as a schema-validated Pydantic model. Each record therefore follows the same nested structure, regardless of the clinical trial design.

Schema Structure

Every field has an explicit default or fallback value when the corresponding information is not stated in the source text. This allows the dataset to distinguish between information that is genuinely absent from the source and information that was missed during extraction.

Annotation Hierarchy

The structured annotations follow the hierarchy below:

Record
├── trial_id
├── title
├── abstract
└── labels
    ├── trial_metadata
    │   ├── registry_source
    │   ├── trial_phase
    │   ├── study_design
    │   ├── blinding
    │   └── years_of_enrollment
    ├── population
    │   ├── disease
    │   └── total_number_of_patients
    ├── arms
    │   ├── arm_name
    │   └── arm_type
    └── outcomes
        ├── abbreviation
        └── expansion

Annotation Methodology

The dataset was manually annotated by a pharmacoeconomic domain expert following a strict non-inferential extraction policy.

Only information explicitly stated in the source title or abstract was annotated. No inference, deduction, or external domain knowledge was used to complete missing fields. When information was not explicitly reported, the corresponding schema default or fallback value was assigned.

This annotation strategy ensures that the dataset distinguishes between:

1. 1.Information explicitly reported in the source.
2. 2.Information not reported in the source.
3. 3.Information that should not be inferred from contextual knowledge.

Expert annotation was selected instead of automated LLM-based annotation because preliminary evaluation showed that LLM-generated labels did not consistently satisfy the non-inferential annotation constraint, particularly for intervention/comparator arms and outcome-level fields.

Trial registry information was also not used as a substitute for the published abstract because registry records cannot always be reliably aligned with the exact version and content of the corresponding publication.

Intended Use

The dataset is intended for research and development in:

• —Pharmacoeconomic evidence extraction
• —Clinical trial information extraction
• —Structured medical information extraction
• —PICO-style information extraction
• —Retrieval-augmented generation (RAG)
• —Domain-specific small language models
• —Schema-constrained text generation
• —Evidence synthesis and decision-support research

Potential applications include:

• —Fine-tuning and evaluating language models for structured extraction from clinical trial abstracts
• —Benchmarking PICO-style extraction approaches
• —Developing retrieval and evidence-synthesis pipelines
• —Evaluating schema-constrained information extraction
• —Developing structured and auditable clinical-trial evidence extraction systems

The dataset was used in the associated study to train and evaluate fine-tuned small language models for pharmacoeconomic evidence extraction.

Data Sources

The records were derived from published clinical trial literature.

The dataset repository does not redistribute the full-text publications. It contains the publication title, abstract text, and expert-produced structured annotations.

The original publications remain subject to their respective copyright and licensing conditions. Users should obtain the original publications through the appropriate publisher, repository, or open-access source where available.

Ethical and Privacy Considerations

The dataset contains information derived from published clinical trial reports and does not intentionally include direct personal identifiers of individual patients.

The dataset is intended for research purposes and should not be used as a substitute for clinical judgment, regulatory assessment, or formal health technology assessment.

Limitations

The dataset has several limitations:

• —It contains 250 records, which limits its statistical power for large-scale benchmarking and may restrict the performance of models trained exclusively on this dataset.
• —The dataset represents the therapeutic area and clinical trial literature investigated in the associated study and may therefore not generalize to all therapeutic areas, diseases, trial designs, or publication formats.
• —The annotations are based on information explicitly available in the title and abstract. Consequently, information available only in the full text, tables, figures, supplementary materials, or trial registries is not necessarily represented.
• —The relatively small dataset size makes it more suitable for domain-specific experimentation and proof-of-concept research than for large-scale general-purpose model training.

Citation

If you use this dataset in your research, please cite the associated publication:

Youssef M, Mashaly M, Ashour M, El-Tayebi H.
Credibility-Aware Pharmacoeconomic Evidence Extraction from Clinical Trials via RAG-Enhanced Fine-Tuned Small Language Models.
[Publication details to be added after publication]

Acknowledgment

The dataset was developed as part of research on credibility-aware, retrieval-augmented, and fine-tuned small language models for pharmacoeconomic evidence extraction from clinical trial literature.