Datasetpublic

jessicalamjh/discussion-generation

PMCOA Discussion Generation Dataset A dataset of 627 biomedical papers from PubMed Central Open Access, built for the task of discussion section generation: given a manuscript (with its Discussion section removed) and the full text of its cited papers, generate the Discussion section. Each sample contains: manuscript — the paper with its Discussion section removed relevant_papers — full text of the papers cited in the gold discussion gold_discussion — the ground-truth… See the full description on the dataset page: https://huggingface.co/datasets/jessicalamjh/discussion-generation.

sourceHugging Faceotherupdated 2mo agoView on Hugging Face

0likes27downloads

Dataset Card

PMCOA Discussion Generation Dataset

A dataset of 627 biomedical papers from PubMed Central Open Access, built for the task of discussion section generation: given a manuscript (with its Discussion section removed) and the full text of its cited papers, generate the Discussion section.

Each sample contains:

—manuscript — the paper with its Discussion section removed
—relevant_papers — full text of the papers cited in the gold discussion
—gold_discussion — the ground-truth Discussion section

The canonical schema is defined in `src/discussion_generation/data/schemas.py` and documented in detail in `src/discussion_generation/data/README.md`.

Why some fields are JSON strings

Apache Arrow (which backs HuggingFace datasets) requires every column to have a fixed, uniform schema. Two patterns in the native schema are incompatible with that:

Pattern	Arrow's problem
`dict[PaperIdType, ...]` — keys are an open `str` enum	Arrow infers column types from the first record; unseen key names in later records break the schema
`list[Content]` where `Content = Paragraph \	Section` — recursive, polymorphic	Arrow cannot represent recursive or union-typed nested structs

The following fields are serialized to JSON strings before upload:

HF column	Native type	Reason
`manuscript`	`Paper`	recursive / polymorphic, plus dynamic dict keys in nested `all_known_paper_ids`/`bibliography`
`relevant_papers`	`dict[BibliographyEntryId, Paper]`	dynamic dict keys at top level
`gold_discussion`	`Section`	recursive / polymorphic

Additionally, ContentId (tuple[int, ...]) is stored as list[int] because Arrow has no tuple type. This affects gold_discussion_content_id and each element of content_ids_referenced_in_gold_discussion.

All other fields keep their original structure.

Restoring the original structure

Parse JSON strings back and validate with the Sample Pydantic model. Pydantic handles list[int] → tuple[int, ...] coercion for ContentId fields automatically.

python

import json
from datasets import load_dataset
from discussion_generation.data.schemas import Sample

ds = load_dataset("jessicalamjh/discussion-generation", split="train")

def restore(record: dict) -> Sample:
    record = dict(record)

    record["manuscript"]      = json.loads(record["manuscript"])
    record["relevant_papers"] = json.loads(record["relevant_papers"])
    record["gold_discussion"] = json.loads(record["gold_discussion"])

    return Sample.model_validate(record)

samples: list[Sample] = [restore(r) for r in ds]

Or, more simply, via load_schematized_dataset in `src/discussion_generation/utils/data.py`, which does the same thing:

python

from discussion_generation.utils.data import load_schematized_dataset

samples = load_schematized_dataset("jessicalamjh/discussion-generation", split="train")