datasets
Training and evaluation data, with the modality, task and licence stated up front. Listed live from the Hugging Face Hub.
arxiv-summarization
Arxiv dataset for summarization
Dataset for summarization of long documents.Adapted from this repo.Note that original data are pre-tokenized so this dataset returns " ".join(text) and add "\n" for paragraphs. This dataset is compatible with the run_summarization.py script from Transformers if you add this line to the summarization_name_mapping variable:
"ccdv/arxiv-summarization": ("article", "abstract")
Data Fields
id: paper id
article: a string containing the body of… See the full description on the dataset page: https://huggingface.co/datasets/ccdv/arxiv-summarization.pubmed-summarization
PubMed dataset for summarization
Dataset for summarization of long documents.Adapted from this repo.Note that original data are pre-tokenized so this dataset returns " ".join(text) and add "\n" for paragraphs. This dataset is compatible with the run_summarization.py script from Transformers if you add this line to the summarization_name_mapping variable:
"ccdv/pubmed-summarization": ("article", "abstract")
Data Fields
id: paper id
article: a string containing the body… See the full description on the dataset page: https://huggingface.co/datasets/ccdv/pubmed-summarization.govreport-summarization
GovReport dataset for summarization
Dataset for summarization of long documents.Adapted from this repo and this paperThis dataset is compatible with the run_summarization.py script from Transformers if you add this line to the summarization_name_mapping variable:
"ccdv/govreport-summarization": ("report", "summary")
Data Fields
id: paper id
report: a string containing the body of the reportsummary: a string containing the summary of the report
Data Splits… See the full description on the dataset page: https://huggingface.co/datasets/ccdv/govreport-summarization.nemotron-cc-v21-Parsed-QA4-Summarization-Qwen3-1.7BNLG-Abstractive-Summarization
SEA Abstractive Summarization
SEA Abstractive Summarization evaluates a model's ability to read a document, identify the key points within, and summarize them into a coherent and fluent text while paraphrasing the document. It is sampled from XL-Sum for Indonesian, Tamil, Thai, and Vietnamese.
Supported Tasks and Leaderboards
SEA Abstractive Summarization is designed for evaluating chat or instruction-tuned large language models (LLMs). It is part of the SEA-HELM… See the full description on the dataset page: https://huggingface.co/datasets/aisingapore/NLG-Abstractive-Summarization.legal_summarization
LegalSummarization
An MTEB dataset
Massive Text Embedding Benchmark
The dataset consistes of 439 pairs of contracts and their summarizations from https://tldrlegal.com and https://tosdr.org/.
Task category
t2t
Domains
Legal, Written
Reference
https://github.com/lauramanor/legal_summarization
How to evaluate on this task
You can evaluate an embedding model on this dataset using the following code:
import mteb
task =… See the full description on the dataset page: https://huggingface.co/datasets/mteb/legal_summarization.long-context-text-summarization-alpaca-formatspeech-translation-and-summarization
English-Centric Multilingual Audio Dataset
This dataset contains generated article and summary audio for English-centric multilingual directions.
Each direction folder contains metadata JSONL files and corresponding audio files for few_shot and test splits.
Included directions
amharic_english / english_amharic
arabic_english / english_arabic
bengali_english / english_bengali
chinese_simplified_english / english_chinese_simplified
english_english
french_english /… See the full description on the dataset page: https://huggingface.co/datasets/McGill-NLP/speech-translation-and-summarization.naver-news-summarization-ko
Naver-News-KO: A Korean News Summarization Dataset
A Korean news summarization dataset of 27,400 (document, summary) pairs, crawled from
Naver News over a ten-day window in July 2022. It was originally built for a
Korean NLP hands-on lab and has been publicly hosted on the Hugging Face Hub since January 2023.
A technical report documenting the collection protocol, corpus statistics, contamination analysis, and
reproducible baselines is available on arXiv: arXiv:2607.20442.… See the full description on the dataset page: https://huggingface.co/datasets/daekeun-ml/naver-news-summarization-ko.VG_summarization
VG Summarization Dataset
The source of this dataset is Norsk Aviskorpus (Norwegian newspaper corpus). This corpus includes articles from Norway’s largest newspaper from 1998 to 2019. In this dataset, we used
the first paragraph (lead) of each article as its summary. This dataset only includes articles from the Norwegian newspaper "VG".
The quality of the summary-article pairs has not been evaluated.
License
Please refer to the license of Norsk Aviskorpus
Citation… See the full description on the dataset page: https://huggingface.co/datasets/navjordj/VG_summarization.nordjylland-news-summarization
Dataset Card for "nordjylland-news-summarization"
Dataset Summary
This dataset consists of pairs containing text and corresponding summaries extracted from the Danish newspaper TV2 Nord.
Supported Tasks and Leaderboards
Summarization is the intended task for this dataset. No leaderboard is active at this point.
Languages
The dataset is available in Danish (da).
Dataset Structure
An example from the dataset looks as… See the full description on the dataset page: https://huggingface.co/datasets/alexandrainst/nordjylland-news-summarization.scielo-summarization
LoRaLay: A Multilingual and Multimodal Dataset for Long Range and Layout-Aware Summarization
A collaboration between reciTAL, MLIA (ISIR, Sorbonne Université), Meta AI, and Università di Trento
SciELO dataset for summarization
SciELO is a dataset for summarization of research papers written in Spanish and Portuguese, for which layout information is provided.
Data Fields
article_id: article id
article_words: sequence of words constituting the body of… See the full description on the dataset page: https://huggingface.co/datasets/nglaura/scielo-summarization.scored_summarization_datasets
Dataset Card for "Scored-Summarization-datasets"
A collection of Text summarization datasets geared towards training a multi-purpose text summarizer.
Each dataset is a parquet file with the following features.
default
text: a string feature. The source document
summary: a string feature. The summary of the document
provenance: a string feature. Information about the sub dataset.
t5_text_token_count: a int64 feature. The number of tokens the text is encoded in.… See the full description on the dataset page: https://huggingface.co/datasets/jordiclive/scored_summarization_datasets.navjordj-SNL_summarization_copyCopy of navjordj/SNL_summarization
arxivlay-summarization
LoRaLay: A Multilingual and Multimodal Dataset for Long Range and Layout-Aware Summarization
A collaboration between reciTAL, MLIA (ISIR, Sorbonne Université), Meta AI, and Università di Trento
Arxiv-Lay dataset for summarization
ArXiv-Lay is an enhanced version of the arXiv summarization dataset, for which layout information is provided.
Data Fields
article_id: article id
article_words: sequence of words constituting the body of the article… See the full description on the dataset page: https://huggingface.co/datasets/nglaura/arxivlay-summarization.legal_summarizationgandalf_summarization
gandalf_summarization
This is a dataset of indirect prompt injections from Gandalf by Lakera, specifically from the Adventure 4 level (link, although note the Adventure may no longer be available in the future).
Note that we might update the dataset occasionally by cleaning the data or adding more samples.
Gandalf and prompt injections
Gandalf is a ChatGPT-powered system that – in this level specifically – tries to summarize the text that is provided by the user.
In… See the full description on the dataset page: https://huggingface.co/datasets/Lakera/gandalf_summarization.EDGAR-CORPUS-Financial-Summarization
EDGAR-CORPUS : 10K Financial Report Summarization
Extracted from SEC EDGAR filings (1993-2020). This dataset enhances financial report summarization by leveraging a hybrid AI model strategy.
Using:
ChatGPT-3.5 Turbo(~70%),
Claude 3.5 (~30% to generate structured, accurate, and concise summaries)
Dataset Composition
Summaries in this dataset are generated using a hybrid AI model strategy, balancing quality and efficiency:ChatGPT-3.5 Turbo (~70%) – Used for structured… See the full description on the dataset page: https://huggingface.co/datasets/kritsadaK/EDGAR-CORPUS-Financial-Summarization.formatted-arxiv-summarizationturkish-structured-summarization-1.5m
Turkish Structured Summarization 1.5M v2
Üç cümlelik kurgusal operasyon kayıtları ve kısa Türkçe özetleri.
Doğrulanmış boyut
Train: 1,470,000
Validation: 15,000
Test: 15,000
Toplam: 1,500,000
Ana görev sütunları: id, document, summary, domain
Provenance
Veri insan mesajlarından, belgelerinden veya web kazımasından alınmamıştır. Tamamı
depodaki üretici koduyla deterministik olarak oluşturulur. Her satırda source_type,
provenance, generator_version… See the full description on the dataset page: https://huggingface.co/datasets/GoktugD/turkish-structured-summarization-1.5m.pubmedlay-summarization
LoRaLay: A Multilingual and Multimodal Dataset for Long Range and Layout-Aware Summarization
A collaboration between reciTAL, MLIA (ISIR, Sorbonne Université), Meta AI, and Università di Trento
PubMed-Lay dataset for summarization
PubMed-Lay is an enhanced version of the PubMed summarization dataset, for which layout information is provided.
Data Fields
article_id: article id
article_words: sequence of words constituting the body of the article… See the full description on the dataset page: https://huggingface.co/datasets/nglaura/pubmedlay-summarization.arxiv_summarization_postprocess
Dataset Card for "arxiv_summarization_postprocess"
More Information needed
legal_case_document_summarization
Dataset Card for LegalCaseDocumentSummarization
Dataset Summary
[More Information Needed]
Supported Tasks and Leaderboards
[More Information Needed]
Languages
[More Information Needed]
Dataset Structure
Data Instances
[More Information Needed]
Data Fields
[More Information Needed]
Data Splits
[More Information Needed]
Dataset Creation
Curation Rationale
[More Information Needed]… See the full description on the dataset page: https://huggingface.co/datasets/joelniklaus/legal_case_document_summarization.hal-summarization
LoRaLay: A Multilingual and Multimodal Dataset for Long Range and Layout-Aware Summarization
A collaboration between reciTAL, MLIA (ISIR, Sorbonne Université), Meta AI, and Università di Trento
HAL dataset for summarization
HAL is a dataset for summarization of research papers written in French, for which layout information is provided.
Data Fields
article_id: article id
article_words: sequence of words constituting the body of the article
article_bboxes:… See the full description on the dataset page: https://huggingface.co/datasets/nglaura/hal-summarization.summarization-polish-summaries-corpuslong-summarization-persian
Dataset Card for Long-Summarization-Persian
Dataset Summary
This dataset was created by crawling all categories of ensani.ir
Supported Tasks and Leaderboards
This dataset can use in Text Summarization Tasks.
Languages
Persian language
Citation Information
Dear researchers,
We’re excited to share this dataset with the community! If you find it useful in your work, please cite our paper (linked below) to acknowledge the effort behind its… See the full description on the dataset page: https://huggingface.co/datasets/zedfum/long-summarization-persian.airoboros-summarizationThis is my adaptation and cleaned version of the Booksum dataset to work with Airoboros by Jon Durbin
huggingface
I created this dataset for the purposes of improving the LLM capabilities with summarization. It's a core feature that I feel many applications rely on, yet we're still relying on older Longformer, RoBERTa, or BART solutions.
This dataset has been altered from the original as follows:
Cleaned up bad formatting, extra quotes at the beginning of summaries, extra line breaks, and… See the full description on the dataset page: https://huggingface.co/datasets/mattpscott/airoboros-summarization.summarizationgovreport-summarization-8192
GovReport Summarization - 8192 tokens
ccdv/govreport-summarization with the changes of:
data cleaned with the clean-text python package
total tokens for each column computed and added in new columns according to the long-t5 tokenizer (done after cleaning)
train info
RangeIndex: 8200 entries, 0 to 8199
Data columns (total 4 columns):
# Column Non-Null Count Dtype
--- ------ -------------- -----
0 report 8200 non-null… See the full description on the dataset page: https://huggingface.co/datasets/pszemraj/govreport-summarization-8192.koreascience-summarization
LoRaLay: A Multilingual and Multimodal Dataset for Long Range and Layout-Aware Summarization
A collaboration between reciTAL, MLIA (ISIR, Sorbonne Université), Meta AI, and Università di Trento
KoreaScience dataset for summarization
KoreaScience is a dataset for summarization of research papers written in Korean, for which layout information is provided.
Data Fields
article_id: article id
article_words: sequence of words constituting the body of the… See the full description on the dataset page: https://huggingface.co/datasets/nglaura/koreascience-summarization.