datasets
Training and evaluation data, with the modality, task and licence stated up front. Listed live from the Hugging Face Hub.
wikipedia-paragraphs
wikipedia-paragraphs
wikipedia-paragraphs is a dataset generated from Wikipedia, designed for natural language processing (NLP) research.
Each entry contains cleaned paragraph text and Wikilink information extracted from a Wikipedia page, along with useful metadata such as categories, templates, and the associated Wikidata QID.
Dataset structure
Configurations
The dataset is organized into multiple configurations, such as enwiki-20260607-v1.2.1.… See the full description on the dataset page: https://huggingface.co/datasets/singletongue/wikipedia-paragraphs.openstax_paragraphsTexbooks from openstax.org with their chapters, abstracts and sections.
Sample:
{
"book_title":"World History Volume 1, to 1500",
"language":"en",
"chapters":[
{
"title":"Preface",
"abstract":"None",
"sections":[
{
"title":"About OpenStax",
"paragraph":"OpenStax is part of Rice University, which is a 501(c)(3) nonprofit..."
},
{
"title":"About OpenStax Resources"… See the full description on the dataset page: https://huggingface.co/datasets/HuggingFaceTB/openstax_paragraphs.ccel-paragraphs
CCEL Paragraphs
Dataset Description
Dataset Summary
This dataset includes all paragraphs from the Christian Classics Ethereal Library. It also includes scripture references extracted from the ThML.
Supported Tasks and Leaderboards
It is expected that this dataset can be used as part of the training pipeline for large language models. In particular, it could be used to create a clustering benchmark by using scripture references as labels.… See the full description on the dataset page: https://huggingface.co/datasets/jncraton/ccel-paragraphs.multilingual-wikipedia-paragraphsparagraphs-co84b
paragraphs > release-640
https://universe.roboflow.com/roboflow-100/paragraphs-co84b
This dataset is part of RF100, an Intel-sponsored initiative to create a new object detection benchmark for model generalizability.
Dataset Summary
Total images: 6063
Train: 4209 images
Validation: 1221 images
Test: 633 images
Classes: 7 (g, h, g1, g3, -, m, n)
Format: YOLOv8 (Ultralytics)
License: CC BY 4.0
Preprocessing
Auto-orientation of pixel data (with EXIF-orientation… See the full description on the dataset page: https://huggingface.co/datasets/LibreYOLO/paragraphs-co84b.tae-data-split-paragraphs
Split Paragraphs Dataset
Split paragraphs data with configs 000-099.
multilingual-cc-news-ir-related-paragraph-pairs
multilingual-cc-news-ir-related-paragraph-pairs
This dataset contains multilingual pairs of related news paragraphs derived from the same source article.
Each pair is intended to serve as a broad positive for contrastive learning, representation learning, and retrieval-oriented training.
Dataset at a glance
Task: paragraph-level relatedness / broad-positive contrastive training
Fields:
text_a: first text in the pair
text_b: second text in the pair
source_url:… See the full description on the dataset page: https://huggingface.co/datasets/hotchpotch/multilingual-cc-news-ir-related-paragraph-pairs.enwiki-paragraphsids-paragraph-commentary-corpusThe data was collected from GitHub using https://github.com/cheop-byeon/RFCRationaleBuilder.
This is our first version of data for RFC Rationale.
Dataset Citation
If you find this dataset useful and include it in your studies, please cite our paper:
@inproceedings{bian2024tell,
title={Tell Me Why: Language Models Help Explain the Rationale Behind Internet Protocol Design},
author={Bian, Jie and Welzl, Michael and Kutuzov, Andrey and Arefyev, Nikolay},
booktitle={2024 IEEE… See the full description on the dataset page: https://huggingface.co/datasets/jiebi/ids-paragraph-commentary-corpus.4B-think-paragraph.stride-train1-test32.k-8.statml-arxiv.qwen3-ids
4B-think-paragraph.stride-train1-test32.k-8.statml-arxiv.qwen3-ids
Thoughts for next-token prediction on k=8 token chunks of JackHsieh/statML-arxiv-40M-20M, generated by
Qwen/Qwen3-4B in thinking mode.
The prompted task: reason about what comes IMMEDIATELY next — the next k=8 tokens after the
cut — and answer with a single unconstrained paragraph of dense reasoning, focused on the
exact state at the cut and what the local grammar, notation, or argument forces next. Both the… See the full description on the dataset page: https://huggingface.co/datasets/JackHsieh/4B-think-paragraph.stride-train1-test32.k-8.statml-arxiv.qwen3-ids.wikipedia-paragraphs
Wikipedia Paragraph Samples
Dataset Description
This dataset contains paragraphs extracted from randomly selected English Wikipedia articles. It provides a diverse sample of Wikipedia content across various topics.
Dataset Details
Name: Wikipedia Paragraph Samples
Version: 1.0
Date Created: 2024-08-20
Language: English
Format: JSONLines
Contents
Each line in the dataset represents a single paragraph and contains two fields:
Title of the Wikipedia… See the full description on the dataset page: https://huggingface.co/datasets/agentlans/wikipedia-paragraphs.wikipedia_paragraphs
Description
This dataset consists of English Wikipedia articles, which first have been split by paragraph breaks and subsequently by spaces. For each resulting token, there is a corresponding binary ner_tag, which is 1 if a token was followed by paragraph break in the original text. There are two deliberate exceptions to this, which can be seen in the dataset generation code:
The text is not split if a paragraph break is preceded by a colon (":"), to avoid lists being separated… See the full description on the dataset page: https://huggingface.co/datasets/mamei16/wikipedia_paragraphs.en_wikipedia_paragraphs4B-think-paragraph.stride-train1-test32.k-8.statml-arxiv.qwen3-ids.kv-tags-explained
4B-think-paragraph.stride-train1-test32.k-8.statml-arxiv.qwen3-ids.kv-tags-explained
A pre-tokenized, tag-wrapped variant of JackHsieh/4B-think-paragraph.stride-train1-test32.k-8.statml-arxiv.qwen3-ids.
The source thoughts are two parts: a <think> block, then a single paragraph of dense
reasoning about the immediate continuation. Only the part after </think> — the paragraph —
becomes the VALUE. The reasoning inside the think block is dropped.
The VALUE is capped at 512 tokens.… See the full description on the dataset page: https://huggingface.co/datasets/JackHsieh/4B-think-paragraph.stride-train1-test32.k-8.statml-arxiv.qwen3-ids.kv-tags-explained.wikipedia-paragraph-embeddings-en-gist-complete
Dataset Summary
Paragraph embeddings for every article in English Wikipedia (not the Simple English version).
Based on wikimedia/wikipedia, 20231101.en.
Embeddings were generated with avsolatorio/GIST-small-Embedding-v0
and are quantized to int8.
You can load the data with the following:
from datasets import load_dataset
ds = load_dataset(path="Abrak/wikipedia-paragraph-embeddings-en-gist-complete", data-dir="20231101.en")
Dataset Structure
The structure of the… See the full description on the dataset page: https://huggingface.co/datasets/Abrak/wikipedia-paragraph-embeddings-en-gist-complete.dfm-paragraphs
Dataset Card for "dfm-paragraphs"
More Information needed
jawiki-paragraphsd1_code_long_paragraphscanbert-paragraph-level-longlen-search-v1-384_512_96wikipedia-first-paragraph
Dataset Description
This dataset contains the first paragraph of cleaned Wikipedia articles in English.
It was obtained by transorming the Wikipedia "20220301.en" dataset as follows:
from datasets import load_dataset
dataset = load_dataset("wikipedia", "20220301.en")["train"]
def get_first_paragraph(example):
example["text"] = example['text'].split('\n\n')[0]
return example
dataset = dataset.map(get_first_paragraph)
Why use this dataset?
The size of the original… See the full description on the dataset page: https://huggingface.co/datasets/abokbot/wikipedia-first-paragraph.paragraph-cited-paperswikipedia-multilingual-ir-related-paragraph-pairs
wikipedia-multilingual-ir-related-paragraph-pairs
This dataset contains multilingual pairs of related Wikipedia paragraphs.
Each pair is sampled from the same article and the same section, so the two paragraphs are topically related but not necessarily paraphrases.
The dataset is intended for large-scale contrastive learning, representation learning, and retrieval-style training where broad positives are useful.
Dataset at a glance
Task: paragraph-level… See the full description on the dataset page: https://huggingface.co/datasets/hotchpotch/wikipedia-multilingual-ir-related-paragraph-pairs.wikipedia-first-paragraphparagraph-diss-genparagraphs-cited-papers-combinedwiki_paragraphs_norwegian
WIKI Paragraphs Norwegian
A multi-split dataset for machine learning research and evaluation, containing text samples in JSON Lines format.
Features
Multiple splits for different use cases
Random shuffle with Fisher-Yates algorithm
Structured format with text and metadata
Size-varied validation/test sets (100 to 10k samples)
Splits Overview
Split Name
Samples
Typical Usage
train
1,000,000
Primary training data
validation
10,000
Standard… See the full description on the dataset page: https://huggingface.co/datasets/pere/wiki_paragraphs_norwegian.wikipedia-short-paragraphsfr_wiki_paragraphs
Dataset Card for French Wikipedia Text Corpus
Dataset Description
The French Wikipedia Text Corpus is a comprehensive dataset derived from French Wikipedia articles. It is specifically designed for training language models (LLMs). The dataset contains the text of paragraphs from Wikipedia articles, with sections, footnotes, and titles removed to provide a clean and continuous text stream.
Dataset Details
Features
text: A single attribute containing the full text of… See the full description on the dataset page: https://huggingface.co/datasets/1ou2/fr_wiki_paragraphs.japanese-wikipedia-paragraphsA slightly modified version of the parsing and chunking method for singletongue/wikipedia-utils.
Pre-processing was performed using oshizo/wikipedia-utils, which is a fork of the original repository, singletongue/wikipedia-utils.
The Wikipedia data was crawled between 2023/12/5 and 2023/12/8.
books3_lowgrade_paragraphs
Dataset Card for "books3_lowgrade_paragraphs"
the_pile books3, books with smog grade difficulty estimate between 6.6 or and 7.1. Split into paragraphs and filtered out most 'non-paragraphs' like titles, tables of content, etc.
For easier books, see books3_basic_paragraphs