EMBO/soda-vec-data-full_pmc_title_abstract

SODA-VEC Clean Dataset

This is a cleaned and filtered version of the SODA-VEC dataset, containing high-quality biomedical title-abstract pairs from PubMed Central (PMC) articles.

Dataset Overview

• —Total examples: 26,573,900
• —Training set: 26,473,900 examples (99.6%)
• —Validation set: 50,000 examples (0.2%)
• —Test set: 50,000 examples (0.2%)

Quality Filtering Applied

This dataset has been processed with the following quality filters:

Abstract Length Filtering

• —Minimum length: 128 characters
• —Maximum length: 6,000 characters
• —Rationale: Removes fragments and overly long texts while preserving scientific abstracts

Retention Statistics

• —Original dataset: ~26.6M examples
• —After filtering: 26,573,900 examples
• —Retention rate: ~99.7%

Content Statistics (sample):

• —Title length: ~100 ± 50 chars
• —Abstract length: ~1,300 ± 600 chars
• —Title range: 10-500 chars
• —Abstract range: 128-6,000 chars

Length Distributions

[image]

The dataset shows well-balanced length distributions:

• —Title lengths: Centered around 100 characters with good variance
• —Abstract lengths: Normally distributed around 1,300 characters
• —Quality filtering: Clearly removes outliers while preserving natural variation

[image]

Data Fields

Each example contains:

• —`title` (string): The title of the scientific article
• —`abstract` (string): The abstract of the scientific article
• —`pmcid` (string): PubMed Central ID for the article

Intended Use

This dataset is designed for:

Primary Applications

• —Scientific text embeddings: Training domain-specific embedding models
• —Biomedical NLP: Fine-tuning language models on scientific literature
• —Semantic similarity: Learning representations for scientific text matching
• —Information retrieval: Building search systems for biomedical literature

Research Applications

• —Representation learning for scientific texts
• —Cross-modal learning (title-abstract relationships)
• —Domain adaptation for biomedical language models
• —Evaluation of scientific text understanding systems

Data Source & Methodology

Original Dataset

Based on the SODA-VEC dataset from PubMed Central articles.

Processing Pipeline

1. 1.Data loading: Combined train/validation/test splits from original dataset
2. 2.Quality filtering: Applied length-based filters to ensure high-quality pairs
3. 3.Split creation: Created new balanced train/validation/test splits
4. 4.Validation: Verified data integrity and distribution balance

Quality Assurance

• —Length distribution analysis
• —Duplicate detection and removal
• —Content quality validation
• —Statistical validation of splits

Usage Example

from datasets import load_dataset

# Load the dataset
dataset = load_dataset("EMBO/soda-vec-data-full_pmc_title_abstract")

# Access different splits
train_data = dataset["train"]
val_data = dataset["validation"] 
test_data = dataset["test"]

# Example usage
for example in train_data.take(1):
    print(f"Title: {example['title']}")
    print(f"Abstract: {example['abstract']}")
    print(f"PMC ID: {example['pmcid']}")

Citation

If you use this dataset in your research, please cite the original SODA-VEC paper:

@article{soda-vec-2024,
  title={SODA-VEC: Training Vector Representations of Scientific Literature},
  author={...},
  journal={...},
  year={2024}
}

License

This dataset is released under the CC-BY-4.0 license, consistent with PubMed Central's open access requirements.

Contact

For questions about this dataset, please contact the EMBO team or open an issue in the dataset repository.

Dataset processed with quality filters and balanced splits for optimal training performance.