datasets
Training and evaluation data, with the modality, task and licence stated up front. Listed live from the Hugging Face Hub.
wikipedia-paragraphs
wikipedia-paragraphs
wikipedia-paragraphs is a dataset generated from Wikipedia, designed for natural language processing (NLP) research.
Each entry contains cleaned paragraph text and Wikilink information extracted from a Wikipedia page, along with useful metadata such as categories, templates, and the associated Wikidata QID.
Dataset structure
Configurations
The dataset is organized into multiple configurations, such as enwiki-20260607-v1.2.1.… See the full description on the dataset page: https://huggingface.co/datasets/singletongue/wikipedia-paragraphs.openstax_paragraphsTexbooks from openstax.org with their chapters, abstracts and sections.
Sample:
{
"book_title":"World History Volume 1, to 1500",
"language":"en",
"chapters":[
{
"title":"Preface",
"abstract":"None",
"sections":[
{
"title":"About OpenStax",
"paragraph":"OpenStax is part of Rice University, which is a 501(c)(3) nonprofit..."
},
{
"title":"About OpenStax Resources"… See the full description on the dataset page: https://huggingface.co/datasets/HuggingFaceTB/openstax_paragraphs.ccel-paragraphs
CCEL Paragraphs
Dataset Description
Dataset Summary
This dataset includes all paragraphs from the Christian Classics Ethereal Library. It also includes scripture references extracted from the ThML.
Supported Tasks and Leaderboards
It is expected that this dataset can be used as part of the training pipeline for large language models. In particular, it could be used to create a clustering benchmark by using scripture references as labels.… See the full description on the dataset page: https://huggingface.co/datasets/jncraton/ccel-paragraphs.enwiki-paragraphswikipedia-paragraphs
Wikipedia Paragraph Samples
Dataset Description
This dataset contains paragraphs extracted from randomly selected English Wikipedia articles. It provides a diverse sample of Wikipedia content across various topics.
Dataset Details
Name: Wikipedia Paragraph Samples
Version: 1.0
Date Created: 2024-08-20
Language: English
Format: JSONLines
Contents
Each line in the dataset represents a single paragraph and contains two fields:
Title of the Wikipedia… See the full description on the dataset page: https://huggingface.co/datasets/agentlans/wikipedia-paragraphs.wikipedia_paragraphs
Description
This dataset consists of English Wikipedia articles, which first have been split by paragraph breaks and subsequently by spaces. For each resulting token, there is a corresponding binary ner_tag, which is 1 if a token was followed by paragraph break in the original text. There are two deliberate exceptions to this, which can be seen in the dataset generation code:
The text is not split if a paragraph break is preceded by a colon (":"), to avoid lists being separated… See the full description on the dataset page: https://huggingface.co/datasets/mamei16/wikipedia_paragraphs.dfm-paragraphs
Dataset Card for "dfm-paragraphs"
More Information needed
jawiki-paragraphsd1_code_long_paragraphsjapanese-wikipedia-paragraphsA slightly modified version of the parsing and chunking method for singletongue/wikipedia-utils.
Pre-processing was performed using oshizo/wikipedia-utils, which is a fork of the original repository, singletongue/wikipedia-utils.
The Wikipedia data was crawled between 2023/12/5 and 2023/12/8.
paragraphs-cited-papers-combinedwiki_paragraphs_norwegian
WIKI Paragraphs Norwegian
A multi-split dataset for machine learning research and evaluation, containing text samples in JSON Lines format.
Features
Multiple splits for different use cases
Random shuffle with Fisher-Yates algorithm
Structured format with text and metadata
Size-varied validation/test sets (100 to 10k samples)
Splits Overview
Split Name
Samples
Typical Usage
train
1,000,000
Primary training data
validation
10,000
Standard… See the full description on the dataset page: https://huggingface.co/datasets/pere/wiki_paragraphs_norwegian.wikipedia-short-paragraphsfr_wiki_paragraphs
Dataset Card for French Wikipedia Text Corpus
Dataset Description
The French Wikipedia Text Corpus is a comprehensive dataset derived from French Wikipedia articles. It is specifically designed for training language models (LLMs). The dataset contains the text of paragraphs from Wikipedia articles, with sections, footnotes, and titles removed to provide a clean and continuous text stream.
Dataset Details
Features
text: A single attribute containing the full text of… See the full description on the dataset page: https://huggingface.co/datasets/1ou2/fr_wiki_paragraphs.books3_lowgrade_paragraphs
Dataset Card for "books3_lowgrade_paragraphs"
the_pile books3, books with smog grade difficulty estimate between 6.6 or and 7.1. Split into paragraphs and filtered out most 'non-paragraphs' like titles, tables of content, etc.
For easier books, see books3_basic_paragraphs
ResearchArcade-openreview-paragraphsarxiv-paragraphsGutenQA_Paragraphs
📚 GutenQA-Paragraphs
GutenQA-Paragraphs consists on the same 100 Public Domain Narrative Books used in GutenQA. In this version, passages are extracted at the paragraph level.
The GutenQA dataset, as available, is the result of applying the text segmentation method LumberChunker to the GutenQA-Paragraphs.
The dataset is organized into the following columns:
Book Name: The title of the book from which the passage is extracted.
Book ID: A unique integer identifier assigned to each… See the full description on the dataset page: https://huggingface.co/datasets/LumberChunker/GutenQA_Paragraphs.jawiki-paragraphs-unsup-simcse-bert-base-japanese-v3
Dataset Card for "jawiki-paragraphs-unsup-simcse-bert-base-japanese-v3"
More Information needed
wiki_paragraphs_english
WIKI Paragraphs English
A multi-split dataset for machine learning research and evaluation, containing text samples in JSON Lines format.
Features
Multiple splits for different use cases
Random shuffle with Fisher-Yates algorithm
Structured format with text and metadata
Size-varied validation/test sets (100 to 10k samples)
Splits Overview
Split Name
Samples
Typical Usage
train
1,000,000
Primary training data
validation
10,000
Standard validation… See the full description on the dataset page: https://huggingface.co/datasets/pere/wiki_paragraphs_english.d1_math_long_paragraphsSynthetic-Pretrain-Paragraphs-150Topics
Synthetic-Pretrain-Paragraphs-150Topics
A synthetic dataset consisting of continuous text paragraphs in Russian and English, generated using Qwen2.5-7B-Instruct.
Dataset Curation
Source: Generated via vLLM on an RTX 3060.
Topics: Covers 150 fundamental fields of knowledge, ranging from programming to speleology.
Constraint: Strict prohibition on greetings, lists, and conversational fillers; contains only raw factual descriptions.
Warning: As pure synthetic data, it… See the full description on the dataset page: https://huggingface.co/datasets/lumasik/Synthetic-Pretrain-Paragraphs-150Topics.wikipedia-paragraphs-complete
Wikipedia Paragraphs Complete Dataset
This dataset consists of English Wikipedia paragraphs ranging from 1 000 to 8 000 characters in length. It was sourced from the Wikimedia dump: "wikimedia/wikipedia", "20231101.en".
Preprocessing Steps
The dataset has undergone extensive cleaning and normalization, including:
Removing brackets
Removing HTML tags
Normalizing bullet points, hyphenated words, quotation marks, Unicode characters, and whitespace
Replacing email… See the full description on the dataset page: https://huggingface.co/datasets/agentlans/wikipedia-paragraphs-complete.wiki-paragraphs
Dataset Card for wiki-paragraphs
Dataset Summary
The wiki-paragraphs dataset is constructed by automatically sampling two paragraphs from a Wikipedia article. If they are from the same section, they will be considered a "semantic match", otherwise as "dissimilar". Dissimilar paragraphs can in theory also be sampled from other documents, but have not shown any improvement in the particular evaluation of the linked work.The alignment is in no way meant as an accurate… See the full description on the dataset page: https://huggingface.co/datasets/dennlinger/wiki-paragraphs.R1-reverse-wikipedia-paragraphs-v1-1000ResearchArcade-arxiv-paragraphsbooks3_basic_paragraphs
Dataset Card for "books3_basic_paragraphs"
the_pile books3, books with smog grade difficulty estimate of 6.5 or under. Split into paragraphs and filtered out most 'non-paragraphs' like titles, tables of content, etc.
ChemRxiv-Paragraphs
ChemRxiv Paragraphs
This dataset consists of paragraphs from ChemRxiv papers with CC BY 4.0 and CC BY-NC 4.0 licenses, sourced from the BASF-AI/ChemRxiv-Papers dataset. Paragraphs are extracted using Grobid, and filtered using an average log word probability method similar to the approach in allenai/peS2o. Paragraphs with fewer than 50 words are excluded.
The number of unique papers in each license category is as follows:
CC BY 4.0: 5,848 papers
CC BY-NC 4.0: 3,082 papers
To… See the full description on the dataset page: https://huggingface.co/datasets/BASF-AI/ChemRxiv-Paragraphs.ndla_parallel_paragraphs
NDLA Parallel Paragraphs
Dataset Summary
This dataset is derived from articles provided through the NDLA (Norwegian Digital Learning Arena) API. It consists of aligned paragraph-level translations between Norwegian Bokmål and Norwegian Nynorsk. The data is sourced from educational articles designed for upper secondary education and has been collected via the official NDLA Article API.
The dataset is intended for machine translation, language modeling, and linguistic… See the full description on the dataset page: https://huggingface.co/datasets/NbAiLab/ndla_parallel_paragraphs.UnitedNations-ParagraphsAlligned-ar-en-dataset