datasets
Training and evaluation data, with the modality, task and licence stated up front. Listed live from the Hugging Face Hub.
wikipedia
Dataset Card for Wikimedia Wikipedia
Dataset Summary
Wikipedia dataset containing cleaned articles of all languages.
The dataset is built from the Wikipedia dumps (https://dumps.wikimedia.org/)
with one subset per language, each containing a single train split.
Each example contains the content of one full Wikipedia article with cleaning to strip
markdown and unwanted sections (references, etc.).
All language subsets have already been processed for recent dump… See the full description on the dataset page: https://huggingface.co/datasets/wikimedia/wikipedia.wikipedia_culturax_dutch
Filtered CulturaX + Wikipedia for Dutch
This is a combined and filtered version of CulturaX and Wikipedia, only including Dutch. It is intended for the training of LLMs.
Different configs are available based on the number of tokens (see a section below with an overview). This can be useful if you want to know exactly how many tokens you have. Great for using as a streaming dataset, too. Tokens are counted as white-space tokens, so depending on your tokenizer, you'll likely end up… See the full description on the dataset page: https://huggingface.co/datasets/BramVanroy/wikipedia_culturax_dutch.wikipediaThis dataset contains preprocessed and chunked Wikipedia HTML dumps from 25 languages.
Refer to the following for more information:
GitHub repository: https://github.com/stanford-oval/WikiChat
Papers:
WikiChat: Stopping the Hallucination of Large Language Model Chatbots by Few-Shot Grounding on Wikipedia
SPAGHETTI: Open-Domain Question Answering from Heterogeneous Data Sources with Retrieval and Semantic Parsing
WikiChat
Stopping the… See the full description on the dataset page: https://huggingface.co/datasets/stanford-oval/wikipedia.wikipedia-monthly
🚀 Wikipedia Monthly
Last updated: March 14, 2026, 21:06 UTC
This repository provides monthly, multilingual dumps of Wikipedia, processed and prepared for easy use in NLP projects.
📊 Current Statistics
Metric
Current Export (March 2026)
All Exports (Total)
Languages
343
361
Articles
62.8M
62.8M
Usage
Load any language with a single line of code using 🤗 datasets.
latest always refers to the most recent dump, while dated configs refer to… See the full description on the dataset page: https://huggingface.co/datasets/omarkamali/wikipedia-monthly.Wikipedia-AbstractWikipedia Abstract
Introducing Wikipedia Abstract, a comprehensive dataset encompassing abstracts, complete articles, and a popularity score index for both widely spoken and lesser-known Wikipedia subsets. Our dedication to Wikipedia-X ensures a centralized Wikipedia dataset that undergoes regular updates and adheres to the highest standards.
A central focus of our efforts was to include exotic languages that often lack up-to-date Wikipedia dumps or may not have any dumps at all.… See the full description on the dataset page: https://huggingface.co/datasets/laion/Wikipedia-Abstract.wikipediaWikipedia dataset containing cleaned articles of all languages.
The datasets are built from the Wikipedia dump
(https://dumps.wikimedia.org/) with one split per language. Each example
contains the content of one full Wikipedia article with cleaning to strip
markdown and unwanted sections (references, etc.).wikipedia-cn-20230720-filtered本数据集基于中文维基2023年7月20日的dump存档。作为一项以数据为中心的工作，本数据集仅保留了 254,547条 质量较高的词条内容。具体而言：
过滤了Template, Category, Wikipedia, File, Topic, Portal, MediaWiki, Draft, Help等特殊类型的词条
使用启发式的方法和自有的NLU模型过滤了一部分质量较低的词条
过滤了一部分内容较为敏感或存在争议性的词条。
进行了简繁转换和习惯用词转换，确保符合中国大陆地区的习惯用词。
This dataset is based on the Chinese Wikipedia dump archive from July 20th, 2023. As a data-centric effort, the dataset retains 254,574 high-quality entries. Specifically:
Entries of special types such as Template, Category, Wikipedia, File, Topic… See the full description on the dataset page: https://huggingface.co/datasets/0xDing/wikipedia-cn-20230720-filtered.wikipedia-20230901.en-deduped
wikipedia - 20230901.en - deduped
purpose: train with less data while maintaining (most) of the quality
This is really more of a "high quality diverse sample" rather than "we are trying to remove literal duplicate documents". Source dataset: graelo/wikipedia.
configs
default
command:
python -m text_dedup.minhash \
--path $ds_name \
--name $dataset_config \
--split $data_split \
--cache_dir "./cache" \
--output $out_dir \
--column $text_column \… See the full description on the dataset page: https://huggingface.co/datasets/BEE-spoke-data/wikipedia-20230901.en-deduped.wikipedia-zh
中文维基百科纯文本数据集（简体中文）
一个条目一条记录，保留章节结构，可直接用于语言模型预训练。
📅 当前版本：2609（维基百科 dump 日期：2026/9/1）
字段
字段
说明
id
维基百科条目 ID
title
条目标题
url
条目原文网址（用未经繁简转换的原始标题组成，直接命中不跳转）
text
条目全文。第一行为标题，章节以 ## 标示，列表为 - ，表格行以 ｜ 分隔，代码用 ``` 围栏，公式保留 LaTeX 并以 $ 包住
不含图说——图说说的是“这张图是什么”，不是条目本身的叙述，它在
图文配对版 与
图文交错版 里。
侧边信息框的事实会抽出来接在文末的 ## 基本资料 章节——生卒年、面积、成立年份
这些往往只写在信息框里，正文不会重复。字段名取自维基自己的模板定义，不是猜的。
没有长度过滤：一句话的条目也是完整的知识内容，要筛随时可以自己做，
被丢掉的救不回来。
2607 以前与 2608 起的差异… See the full description on the dataset page: https://huggingface.co/datasets/yuhuanstudio/wikipedia-zh.simple_wikipedia
simple wikipedia
the 'simple' split of Wikipedia, from Sept 1 2023. The train split contains about 65M tokens,
Pulled via:
dataset = load_dataset(
"wikipedia", language="simple", date="20230901", beam_runner="DirectRunner"
)
stats
train split
general info
<class 'pandas.core.frame.DataFrame'>
RangeIndex: 226242 entries, 0 to 226241
Data columns (total 4 columns):
# Column Non-Null Count Dtype
--- ------ -------------- -----
0 id… See the full description on the dataset page: https://huggingface.co/datasets/pszemraj/simple_wikipedia.wikipedia-zh-tw
中文維基百科純文字資料集（台灣正體中文）
一個條目一筆記錄，保留章節結構，可直接用於語言模型預訓練。
📅 目前版本：2609（維基百科 dump 日期：2026/9/1）
欄位
欄位
說明
id
維基百科條目 ID
title
條目標題
url
條目原文網址（用未經繁簡轉換的原始標題組成，直接命中不轉址）
text
條目全文。第一行為標題，章節以 ## 標示，列表為 - ，表格列以 ｜ 分隔，程式碼用 ``` 圍欄，公式保留 LaTeX 並以 $ 包住
不含圖說——圖說說的是「這張圖是什麼」，不是條目本身的敘述，它在
圖文配對版 與
圖文交錯版 裡。
側邊資訊框的事實會抽出來接在文末的 ## 基本資料 章節——生卒年、面積、成立年份
這些往往只寫在資訊框裡，正文不會重複。欄位名取自維基自己的模板定義，不是猜的。
沒有長度過濾：一句話的條目也是完整的知識內容，要篩隨時可以自己做，
被丟掉的救不回來。
2607 以前與 2608 起的差異… See the full description on the dataset page: https://huggingface.co/datasets/yuhuanstudio/wikipedia-zh-tw.wikipedia
Plain text of Wikipedia
Dataset Description
Size
Example use (python)
Data fields
Notes on data formatting
License
Aknowledgements
Citation
Dataset Description
This dataset is a plain text version of pages from wikipedia.org spaces for several languages
(English,
German,
French,
Spanish,
Italian).
The text is without HTML tags nor wiki templates.
It just includes markdown syntax for headers, lists and tables.
See Notes on data formatting for more details.
It was… See the full description on the dataset page: https://huggingface.co/datasets/OpenLLM-France/wikipedia.zh-tw-wikipedia
台灣正體中文維基百科 (zh-tw Wikipedia)
截至 2023 年 5 月，中文維基百科 2,533,212 篇條目的台灣正體文字內容。每篇條目為一列 (row)，包含 HTML 以及 Markdown 兩種格式。
A nearly-complete collection of 2,533,212 Traditional Chinese (zh-tw) Wikipedia pages, gathered between May 1, 2023, and May 7, 2023. Includes both the original HTML format and an auto-converted Markdown version, which has been processed using vinta/pangu.py.
於 2023 年 5 月 1 日至 5 月 7 日間取自維基百科 action=query & prop=extracts API，內容皆與維基百科網站之台灣正體版本一致，沒有繁簡體混雜的問題。
For development… See the full description on the dataset page: https://huggingface.co/datasets/zetavg/zh-tw-wikipedia.wikipedia-ja-20230101
range3/wikipedia-ja-20230101
This dataset consists of a parquet file from the wikipedia dataset with only Japanese data extracted. It is generated by the following python code.
このデータセットは、wikipediaデータセットの日本語データのみを抽出したparquetファイルで構成されます。以下のpythonコードによって生成しています。
import datasets
dss = datasets.load_dataset(
"wikipedia",
language="ja",
date="20230101",
beam_runner="DirectRunner",
)
for split,ds in dss.items():
ds.to_parquet(f"wikipedia-ja-20230101/{split}.parquet")
Wikipedia-EN-FA-Accessibility-Bridge
Wikipedia EN-FA Accessibility Bridge
Current, attributable English and Persian Wikipedia article snapshots for pages
created during a 69-day recency window, plus an EN↔FA
counterpart index and static accessibility signals.
The reproducible full baseline is the official 2026-08-01 Wikimedia dump:
6,289,549 English articles without Persian, 129,821 Persian
articles without English, and 22,277,907 namespace-0 pages in the
combined parity index. Redirects are retained in the parity… See the full description on the dataset page: https://huggingface.co/datasets/Reza2kn/Wikipedia-EN-FA-Accessibility-Bridge.wikipedia
Dataset Card for Wikimedia Wikipedia
Dataset Summary
Wikipedia dataset containing cleaned articles of all languages.
The dataset is built from the Wikipedia dumps (https://dumps.wikimedia.org/)
with one subset per language, each containing a single train split.
Each example contains the content of one full Wikipedia article with cleaning to strip
markdown and unwanted sections (references, etc.).
All language subsets have already been processed for recent dump, and you… See the full description on the dataset page: https://huggingface.co/datasets/anisafifi/wikipedia.wikipedia-viewerwikipedia dataset, now with viewer enabled! :D
Dataset Card for Wikipedia
Dataset Summary
Wikipedia dataset containing cleaned articles of all languages.
The datasets are built from the Wikipedia dump
(https://dumps.wikimedia.org/) with one split per language. Each example
contains the content of one full Wikipedia article with cleaning to strip
markdown and unwanted sections (references, etc.).
The articles are parsed using the mwparserfromhell tool, which can be… See the full description on the dataset page: https://huggingface.co/datasets/GGUFGuy/wikipedia-viewer.wikipedia
Dataset Card for Wikimedia Wikipedia
Dataset Summary
Wikipedia dataset containing cleaned articles of all languages.
The dataset is built from the Wikipedia dumps (https://dumps.wikimedia.org/)
with one subset per language, each containing a single train split.
Each example contains the content of one full Wikipedia article with cleaning to strip
markdown and unwanted sections (references, etc.).
All language subsets have already been processed for recent dump… See the full description on the dataset page: https://huggingface.co/datasets/knightcy/wikipedia.wikipedia
Dataset Card for Wikimedia Wikipedia
Dataset Summary
Wikipedia dataset containing cleaned articles of all languages.
The dataset is built from the Wikipedia dumps (https://dumps.wikimedia.org/)
with one subset per language, each containing a single train split.
Each example contains the content of one full Wikipedia article with cleaning to strip
markdown and unwanted sections (references, etc.).
All language subsets have already been processed for recent dump… See the full description on the dataset page: https://huggingface.co/datasets/jessteru/wikipedia.wikipedia
Dataset Card for Wikimedia Wikipedia
Dataset Summary
Wikipedia dataset containing cleaned articles of all languages.
The dataset is built from the Wikipedia dumps (https://dumps.wikimedia.org/)
with one subset per language, each containing a single train split.
Each example contains the content of one full Wikipedia article with cleaning to strip
markdown and unwanted sections (references, etc.).
All language subsets have already been processed for recent dump… See the full description on the dataset page: https://huggingface.co/datasets/AspectRS/wikipedia.wikipedia-full
Dataset Card for Wikimedia Wikipedia
Dataset Summary
Wikipedia dataset containing cleaned articles of all languages.
The dataset is built from the Wikipedia dumps (https://dumps.wikimedia.org/)
with one subset per language, each containing a single train split.
Each example contains the content of one full Wikipedia article with cleaning to strip
markdown and unwanted sections (references, etc.).
All language subsets have already been processed for recent dump, and you… See the full description on the dataset page: https://huggingface.co/datasets/akahana/wikipedia-full.wikipedia
Dataset Card for Wikimedia Wikipedia
Dataset Summary
Wikipedia dataset containing cleaned articles of all languages.
The dataset is built from the Wikipedia dumps (https://dumps.wikimedia.org/)
with one subset per language, each containing a single train split.
Each example contains the content of one full Wikipedia article with cleaning to strip
markdown and unwanted sections (references, etc.).
All language subsets have already been processed for recent dump… See the full description on the dataset page: https://huggingface.co/datasets/rancper/wikipedia.wikipedia-paragraphs
Wikipedia Paragraph Samples
Dataset Description
This dataset contains paragraphs extracted from randomly selected English Wikipedia articles. It provides a diverse sample of Wikipedia content across various topics.
Dataset Details
Name: Wikipedia Paragraph Samples
Version: 1.0
Date Created: 2024-08-20
Language: English
Format: JSONLines
Contents
Each line in the dataset represents a single paragraph and contains two fields:
Title of the Wikipedia… See the full description on the dataset page: https://huggingface.co/datasets/agentlans/wikipedia-paragraphs.wikipedia
Dataset Card for Wikimedia Wikipedia
Dataset Summary
Wikipedia dataset containing cleaned articles of all languages.
The dataset is built from the Wikipedia dumps (https://dumps.wikimedia.org/)
with one subset per language, each containing a single train split.
Each example contains the content of one full Wikipedia article with cleaning to strip
markdown and unwanted sections (references, etc.).
All language subsets have already been processed for recent dump, and you… See the full description on the dataset page: https://huggingface.co/datasets/chris951027/wikipedia.wikipedia-zh-mnbvc
zhwiki-mnbvc
分项目：爬取并处理中文维基百科语料
数据时间：202302-202305 （持续更新）
主项目：MNBVC(Massive Never-ending BT Vast Chinese corpus)超大规模中文语料集 https://github.com/esbatmop/MNBVC
该项目清洗流程主要参考：https://kexue.fm/archives/4176/comment-page-1
并且使用组员开发的去重工具进行数据格式化。
总行数（样本）: 10,754,146
一个示例：
{
"文件名": "cleaned/zhwiki-20230420/folder_0/723712.txt",
"是否待查文件": false,
"是否重复文件": false,
"文件大小": 558,
"simhash": 14363740497821204542,
"最长段落长度": 142,
"段落数": 6,
"去重段落数": 6,
"低质量段落数": 0,
"段落": [
{… See the full description on the dataset page: https://huggingface.co/datasets/wanng/wikipedia-zh-mnbvc.wikipedia_en
Wikipedia
Source: https://huggingface.co/datasets/wikipedia
Num examples: 6,623,239
Language: English
from datasets import load_dataset
load_dataset("tdtunlp/wikipedia_en")
wikipedia-monthly
🚀 Wikipedia Monthly
Last updated: March 14, 2026, 21:06 UTC
This repository provides monthly, multilingual dumps of Wikipedia, processed and prepared for easy use in NLP projects.
📊 Current Statistics
Metric
Current Export (March 2026)
All Exports (Total)
Languages
343
361
Articles
62.8M
62.8M
Usage
Load any language with a single line of code using 🤗 datasets.
latest always refers to the most recent dump, while dated configs refer to… See the full description on the dataset page: https://huggingface.co/datasets/wannaphong/wikipedia-monthly.zh-tw-wikipedia
台灣正體中文維基百科 (zh-tw Wikipedia)
截至 2023 年 5 月，中文維基百科 2,533,212 篇條目的台灣正體文字內容。每篇條目為一列 (row)，包含 HTML 以及 Markdown 兩種格式。
A nearly-complete collection of 2,533,212 Traditional Chinese (zh-tw) Wikipedia pages, gathered between May 1, 2023, and May 7, 2023. Includes both the original HTML format and an auto-converted Markdown version, which has been processed using vinta/pangu.py.
於 2023 年 5 月 1 日至 5 月 7 日間取自維基百科 action=query & prop=extracts API，內容皆與維基百科網站之台灣正體版本一致，沒有繁簡體混雜的問題。
For development… See the full description on the dataset page: https://huggingface.co/datasets/sirzmkk/zh-tw-wikipedia.wikipedia
Dataset Card for Wikimedia Wikipedia
Dataset Summary
Wikipedia dataset containing cleaned articles of all languages.
The dataset is built from the Wikipedia dumps (https://dumps.wikimedia.org/)
with one subset per language, each containing a single train split.
Each example contains the content of one full Wikipedia article with cleaning to strip
markdown and unwanted sections (references, etc.).
All language subsets have already been processed for recent dump, and you… See the full description on the dataset page: https://huggingface.co/datasets/Dragnoz/wikipedia.wikipedia-markdown
Marin Markdownified Wikipedia
Markdownified Wikipedia is a large-scale, pre-processed version of the English Wikipedia Enterprise HTML dump consisting of 8.59B tokens. The corpus has been converted to clean, section-aware Markdown for language-model training.
Value
Tokens
8 587 224 558
Primary source
https://dumps.wikimedia.org/other/enterprise_html/runs/20241201/enwiki-NS0-20241201-ENTERPRISE-HTML.json.tar.gz
File format
JSONL
License
CC-BY-SA 4.0 (mirrors… See the full description on the dataset page: https://huggingface.co/datasets/marin-community/wikipedia-markdown.