yilmazzey/arxiv_summarization_20k_preprocessed

ArXiv Summarization Dataset - 20K Preprocessed

A preprocessed dataset of 20,000 ArXiv papers with their full articles and abstracts, designed for abstract generation and summarization tasks.

Dataset Description

This dataset contains 20,000 ArXiv papers that have been filtered and preprocessed to ensure quality for training summarization models. Each example contains the full article text and its corresponding abstract.

Dataset Structure

The dataset has the following structure:

• —article: The full text of the ArXiv paper
• —abstract: The abstract/summary of the paper

Dataset Statistics

• —Total Papers: 20,000
• —Article Word Count:
• —Mean: 5,875.84 words
• —Median: 5,217 words
• —Range: 2,000 - 14,998 words
• —Abstract Word Count:
• —Mean: 179.86 words
• —Median: 166 words
• —Range: 50 - 500 words
• —Length Ratio (article/abstract):
• —Mean: 36.00
• —Median: 32.43
• —Range: 5.01 - 99.98

Filtering Criteria

The dataset was filtered using the following criteria:

• —Minimum article words: 2,000
• —Maximum article words: 15,000
• —Minimum abstract words: 50
• —Maximum abstract words: 500
• —Minimum length ratio (article/abstract): 5
• —Maximum length ratio (article/abstract): 100

Usage

from datasets import load_dataset

# Load the dataset
dataset = load_dataset("yilmazzey/arxiv_summarization_20k_preprocessed")

# Access the data
print(dataset['train'][0])
# Output: {'article': '...', 'abstract': '...'}

Use Cases

This dataset is suitable for:

• —Training abstract generation models
• —Fine-tuning language models for summarization
• —Research on long-form text summarization
• —Evaluating summarization metrics (ROUGE, BLEU, etc.)

Citation

If you use this dataset, please cite:

@dataset{arxiv_summarization_20k_preprocessed,
  title={ArXiv Summarization Dataset - 20K Preprocessed},
  author={Yilmaz, Zeynep},
  year={2024},
  url={https://huggingface.co/datasets/yilmazzey/arxiv_summarization_20k_preprocessed}
}