datasets
Training and evaluation data, with the modality, task and licence stated up front. Listed live from the Hugging Face Hub.
wikitext
Dataset Card for "wikitext"
Dataset Summary
The WikiText language modeling dataset is a collection of over 100 million tokens extracted from the set of verified
Good and Featured articles on Wikipedia. The dataset is available under the Creative Commons Attribution-ShareAlike License.
Compared to the preprocessed version of Penn Treebank (PTB), WikiText-2 is over 2 times larger and WikiText-103 is over
110 times larger. The WikiText dataset also features a far… See the full description on the dataset page: https://huggingface.co/datasets/Salesforce/wikitext.wikipedia
Dataset Card for Wikimedia Wikipedia
Dataset Summary
Wikipedia dataset containing cleaned articles of all languages.
The dataset is built from the Wikipedia dumps (https://dumps.wikimedia.org/)
with one subset per language, each containing a single train split.
Each example contains the content of one full Wikipedia article with cleaning to strip
markdown and unwanted sections (references, etc.).
All language subsets have already been processed for recent dump… See the full description on the dataset page: https://huggingface.co/datasets/wikimedia/wikipedia.wikipediaWikipedia dataset containing cleaned articles of all languages.
The datasets are built from the Wikipedia dump
(https://dumps.wikimedia.org/) with one split per language. Each example
contains the content of one full Wikipedia article with cleaning to strip
markdown and unwanted sections (references, etc.).wiki_dprThis is the wikipedia split used to evaluate the Dense Passage Retrieval (DPR) model.
It contains 21M passages from wikipedia along with their DPR embeddings.
The wikipedia articles were split into multiple, disjoint text blocks of 100 words as passages.wikipedia_culturax_dutch
Filtered CulturaX + Wikipedia for Dutch
This is a combined and filtered version of CulturaX and Wikipedia, only including Dutch. It is intended for the training of LLMs.
Different configs are available based on the number of tokens (see a section below with an overview). This can be useful if you want to know exactly how many tokens you have. Great for using as a streaming dataset, too. Tokens are counted as white-space tokens, so depending on your tokenizer, you'll likely end up… See the full description on the dataset page: https://huggingface.co/datasets/BramVanroy/wikipedia_culturax_dutch.wikipedia-monthly
🚀 Wikipedia Monthly
Last updated: March 14, 2026, 21:06 UTC
This repository provides monthly, multilingual dumps of Wikipedia, processed and prepared for easy use in NLP projects.
📊 Current Statistics
Metric
Current Export (March 2026)
All Exports (Total)
Languages
343
361
Articles
62.8M
62.8M
Usage
Load any language with a single line of code using 🤗 datasets.
latest always refers to the most recent dump, while dated configs refer to… See the full description on the dataset page: https://huggingface.co/datasets/omarkamali/wikipedia-monthly.wikipediaThis dataset contains preprocessed and chunked Wikipedia HTML dumps from 25 languages.
Refer to the following for more information:
GitHub repository: https://github.com/stanford-oval/WikiChat
Papers:
WikiChat: Stopping the Hallucination of Large Language Model Chatbots by Few-Shot Grounding on Wikipedia
SPAGHETTI: Open-Domain Question Answering from Heterogeneous Data Sources with Retrieval and Semantic Parsing
WikiChat
Stopping the… See the full description on the dataset page: https://huggingface.co/datasets/stanford-oval/wikipedia.wikisource
Dataset Card for Wikimedia Wikisource
Dataset Summary
Wikisource dataset containing cleaned articles of all languages.
The dataset is built from the Wikisource dumps (https://dumps.wikimedia.org/)
with one subset per language, each containing a single train split.
Each example contains the content of one full Wikisource text with cleaning to strip
markdown and unwanted sections (references, etc.).
All language subsets have already been processed for recent dump, and you… See the full description on the dataset page: https://huggingface.co/datasets/wikimedia/wikisource.Wikipedia-AbstractWikipedia Abstract
Introducing Wikipedia Abstract, a comprehensive dataset encompassing abstracts, complete articles, and a popularity score index for both widely spoken and lesser-known Wikipedia subsets. Our dedication to Wikipedia-X ensures a centralized Wikipedia dataset that undergoes regular updates and adheres to the highest standards.
A central focus of our efforts was to include exotic languages that often lack up-to-date Wikipedia dumps or may not have any dumps at all.… See the full description on the dataset page: https://huggingface.co/datasets/laion/Wikipedia-Abstract.wikimedia_filtered
Wikimedia
Description
Official Wikimedia wikis are released under a CC BY-SA license. We downloaded the official database dumps from March 2025 of the English-language wikis that are directly managed by the Wikimedia Foundation. These database dumps include the wikitext—MediaWiki’s custom markup language—for each page as well as talk pages, where editors discuss changes made to a page. We only use the most recent version of each page. We converted wikitext to plain text… See the full description on the dataset page: https://huggingface.co/datasets/common-pile/wikimedia_filtered.wikipediaWikipedia dataset containing cleaned articles of all languages.
The datasets are built from the Wikipedia dump
(https://dumps.wikimedia.org/) with one split per language. Each example
contains the content of one full Wikipedia article with cleaning to strip
markdown and unwanted sections (references, etc.).wikipedia-cn-20230720-filtered本数据集基于中文维基2023年7月20日的dump存档。作为一项以数据为中心的工作，本数据集仅保留了 254,547条 质量较高的词条内容。具体而言：
过滤了Template, Category, Wikipedia, File, Topic, Portal, MediaWiki, Draft, Help等特殊类型的词条
使用启发式的方法和自有的NLU模型过滤了一部分质量较低的词条
过滤了一部分内容较为敏感或存在争议性的词条。
进行了简繁转换和习惯用词转换，确保符合中国大陆地区的习惯用词。
This dataset is based on the Chinese Wikipedia dump archive from July 20th, 2023. As a data-centric effort, the dataset retains 254,574 high-quality entries. Specifically:
Entries of special types such as Template, Category, Wikipedia, File, Topic… See the full description on the dataset page: https://huggingface.co/datasets/0xDing/wikipedia-cn-20230720-filtered.wiki_snippets
Dataset Card for "wiki_snippets"
Dataset Summary
Wikipedia version split into plain text snippets for dense semantic indexing.
Supported Tasks and Leaderboards
More Information Needed
Languages
More Information Needed
Dataset Structure
We show detailed information for 2 configurations of the dataset (with 100 snippet passage length and 0 overlap) in
English:
wiki40b_en_100_0: Wiki-40B
wikipedia_en_100_0: Wikipedia
Data Instances… See the full description on the dataset page: https://huggingface.co/datasets/community-datasets/wiki_snippets.wikipedia-20230901.en-deduped
wikipedia - 20230901.en - deduped
purpose: train with less data while maintaining (most) of the quality
This is really more of a "high quality diverse sample" rather than "we are trying to remove literal duplicate documents". Source dataset: graelo/wikipedia.
configs
default
command:
python -m text_dedup.minhash \
--path $ds_name \
--name $dataset_config \
--split $data_split \
--cache_dir "./cache" \
--output $out_dir \
--column $text_column \… See the full description on the dataset page: https://huggingface.co/datasets/BEE-spoke-data/wikipedia-20230901.en-deduped.wikipedia-zh
中文维基百科纯文本数据集（简体中文）
一个条目一条记录，保留章节结构，可直接用于语言模型预训练。
📅 当前版本：2609（维基百科 dump 日期：2026/9/1）
字段
字段
说明
id
维基百科条目 ID
title
条目标题
url
条目原文网址（用未经繁简转换的原始标题组成，直接命中不跳转）
text
条目全文。第一行为标题，章节以 ## 标示，列表为 - ，表格行以 ｜ 分隔，代码用 ``` 围栏，公式保留 LaTeX 并以 $ 包住
不含图说——图说说的是“这张图是什么”，不是条目本身的叙述，它在
图文配对版 与
图文交错版 里。
侧边信息框的事实会抽出来接在文末的 ## 基本资料 章节——生卒年、面积、成立年份
这些往往只写在信息框里，正文不会重复。字段名取自维基自己的模板定义，不是猜的。
没有长度过滤：一句话的条目也是完整的知识内容，要筛随时可以自己做，
被丢掉的救不回来。
2607 以前与 2608 起的差异… See the full description on the dataset page: https://huggingface.co/datasets/yuhuanstudio/wikipedia-zh.wikiteam
Wikiteam
Description
There are many wikis on the internet that are not managed by the Wikimedia foundation, but do use their MediaWiki software to power their wiki.
Many of these wikis have been archived by wikiteam, a collection of volunteers that create unofficial database dumps of wikis and upload them to the Internet Archive.
We download all dumps made by wikiteam when the metadata indicates the wiki was licensed under CC BY, CC BY-SA, or released into the public… See the full description on the dataset page: https://huggingface.co/datasets/common-pile/wikiteam.simple_wikipedia
simple wikipedia
the 'simple' split of Wikipedia, from Sept 1 2023. The train split contains about 65M tokens,
Pulled via:
dataset = load_dataset(
"wikipedia", language="simple", date="20230901", beam_runner="DirectRunner"
)
stats
train split
general info
<class 'pandas.core.frame.DataFrame'>
RangeIndex: 226242 entries, 0 to 226241
Data columns (total 4 columns):
# Column Non-Null Count Dtype
--- ------ -------------- -----
0 id… See the full description on the dataset page: https://huggingface.co/datasets/pszemraj/simple_wikipedia.bhasha-wiki
Dataset Card for Bhasha-Wiki
Translated wikipedia articles
Dataset Details
Dataset is being updated
Dataset Description
We have translated 6.4 million English wikipedia articles into 6 Indic languages. The translations were done using IndicTrans2 model.
Curated by: Soket AI labs
Language(s) (NLP): Hindi, Bengali, Gujarati, Tamil, Kannada, Urdu
License: cc-by-sa-3.0
Uses
For pretraining or Fine tuning for Indic language models
Dataset… See the full description on the dataset page: https://huggingface.co/datasets/soketlabs/bhasha-wiki.wikipedia-zh-tw
中文維基百科純文字資料集（台灣正體中文）
一個條目一筆記錄，保留章節結構，可直接用於語言模型預訓練。
📅 目前版本：2609（維基百科 dump 日期：2026/9/1）
欄位
欄位
說明
id
維基百科條目 ID
title
條目標題
url
條目原文網址（用未經繁簡轉換的原始標題組成，直接命中不轉址）
text
條目全文。第一行為標題，章節以 ## 標示，列表為 - ，表格列以 ｜ 分隔，程式碼用 ``` 圍欄，公式保留 LaTeX 並以 $ 包住
不含圖說——圖說說的是「這張圖是什麼」，不是條目本身的敘述，它在
圖文配對版 與
圖文交錯版 裡。
側邊資訊框的事實會抽出來接在文末的 ## 基本資料 章節——生卒年、面積、成立年份
這些往往只寫在資訊框裡，正文不會重複。欄位名取自維基自己的模板定義，不是猜的。
沒有長度過濾：一句話的條目也是完整的知識內容，要篩隨時可以自己做，
被丟掉的救不回來。
2607 以前與 2608 起的差異… See the full description on the dataset page: https://huggingface.co/datasets/yuhuanstudio/wikipedia-zh-tw.ru-fandom-wiki
d0rj/ru-fandom-wiki
Description
A set of texts collected from the most popular Russian-language fandoms (65 fandoms) on fandom.com.
The dump given on 25.10.2024-27.10.2024 collected using trafilatura library. All texts are in markdown format.
License
The license supports the license text on the source site - Creative Commons Attribution-ShareAlike 3.0 (Unported) (CC-BY-SA).
wikipedia
Plain text of Wikipedia
Dataset Description
Size
Example use (python)
Data fields
Notes on data formatting
License
Aknowledgements
Citation
Dataset Description
This dataset is a plain text version of pages from wikipedia.org spaces for several languages
(English,
German,
French,
Spanish,
Italian).
The text is without HTML tags nor wiki templates.
It just includes markdown syntax for headers, lists and tables.
See Notes on data formatting for more details.
It was… See the full description on the dataset page: https://huggingface.co/datasets/OpenLLM-France/wikipedia.llm-jp-corpus-v4-ja_wiki
llm-jp-corpus-v4 — ja_wiki
Mirror of the ja/ja_wiki sub-corpus of LLM-jp Corpus v4,
built by the LLM-jp Corpus Building WG (NII).
Source: https://gitlab.llm-jp.nii.ac.jp/datasets/llm-jp-corpus-v4
Sub-corpus: ja_wiki
Files: 6 × jsonl.gz (1.9 GB compressed)
Format: one JSON object per line, with a text key and a meta key
(document id, URL, and other provenance fields).
Directory layout mirrors the upstream repository.
License
CC BY-SA 3.0 — inherited from the… See the full description on the dataset page: https://huggingface.co/datasets/Podtech/llm-jp-corpus-v4-ja_wiki.wikimedia
Wikimedia
Description
Official Wikimedia wikis are released under a CC BY-SA license.
We downloaded the official database dumps from March 2025 of the English-language wikis that are directly managed by the Wikimedia foundation.
These database dumps include the wikitext—Mediawiki’s custom markup language—for each page as well as talk pages, where editors discuss changes made for a page.
We only use the most recent version of each page.
We converted wikitext to plain… See the full description on the dataset page: https://huggingface.co/datasets/common-pile/wikimedia.wikimedia
Dataset Card
This dataset is a curated collection of Wikimedia pages in markdown format,
compiled from various Wikimedia projects across multiple languages.
Covered Wikimedia Projects:
wikipedia
wikibooks
wikinews
wikiquote
wikisource
wikiversity
wikivoyage
wiktionary
Supported Languages:
ar (Arabic)
br (Breton)
ca (Catalan)
co (Corsican)
de (German)
en (English)
es (Spanish)
eu (Basque)
fr (French)
frp (Arpitan)
it (Italian)
nl (Dutch)
oc (Occitan)
pcd (Picard)
pt (Portuguese)… See the full description on the dataset page: https://huggingface.co/datasets/OpenLLM-France/wikimedia.zh-tw-wikipedia
台灣正體中文維基百科 (zh-tw Wikipedia)
截至 2023 年 5 月，中文維基百科 2,533,212 篇條目的台灣正體文字內容。每篇條目為一列 (row)，包含 HTML 以及 Markdown 兩種格式。
A nearly-complete collection of 2,533,212 Traditional Chinese (zh-tw) Wikipedia pages, gathered between May 1, 2023, and May 7, 2023. Includes both the original HTML format and an auto-converted Markdown version, which has been processed using vinta/pangu.py.
於 2023 年 5 月 1 日至 5 月 7 日間取自維基百科 action=query & prop=extracts API，內容皆與維基百科網站之台灣正體版本一致，沒有繁簡體混雜的問題。
For development… See the full description on the dataset page: https://huggingface.co/datasets/zetavg/zh-tw-wikipedia.wikipedia-ja-20230101
range3/wikipedia-ja-20230101
This dataset consists of a parquet file from the wikipedia dataset with only Japanese data extracted. It is generated by the following python code.
このデータセットは、wikipediaデータセットの日本語データのみを抽出したparquetファイルで構成されます。以下のpythonコードによって生成しています。
import datasets
dss = datasets.load_dataset(
"wikipedia",
language="ja",
date="20230101",
beam_runner="DirectRunner",
)
for split,ds in dss.items():
ds.to_parquet(f"wikipedia-ja-20230101/{split}.parquet")
sea_wikiSoutheast Asia Wikipedia Data Repository contains Wikipedia Data from Wikipedia HF that focuses
on extraction in all available Languanges and Local Languages across South East Asia, which some of them
are considered as low-resource languages or extremely low-resource languagesscicode_wiki
SciCode Wikipedia Concept Corpus
中文说明
这个数据集是围绕 SciCode benchmark 构造的 Wikipedia Markdown 知识语料。
数据分为五个相互独立的部分：
原始 seed 概念集：每道题的 core_concepts 和 adjacent_concepts。
首轮 expanded 概念集：使用 deepseek-v4-flash 为每道题扩展 25 个概念。
2026-06-12 expanded 概念集：综合 seed 概念和首轮扩展结果，使用
gpt-5.5 为每道题继续扩展 70 个不重复概念。
2026-07-02 expanded 概念集：综合此前所有概念，使用
deepseek-v4-pro 为每道题继续扩展 50 个更发散的概念，并完成 Wikipedia 抓取。
2026-07-02 partial expanded 概念集：使用 deepseek-v4-pro 继续扩展后，
上传截至 2026-07-03 16:16:26 +08:00… See the full description on the dataset page: https://huggingface.co/datasets/dabingzz/scicode_wiki.wikidata-truthy
Wikidata Truthy
Dataset Description
Core facts from Wikidata (preferred statements only)
Original Source: https://dumps.wikimedia.org/wikidatawiki/entities/latest-truthy.nt.bz2
Dataset Summary
This dataset contains RDF triples from Wikidata Truthy converted to HuggingFace
dataset format for easy use in machine learning pipelines.
Format: Originally ntriples, converted to HuggingFace Dataset
Size: 100.0 GB (extracted)
Entities: ~100M
Triples: ~2B
Original… See the full description on the dataset page: https://huggingface.co/datasets/CleverThis/wikidata-truthy.Wiki_FR_2026.07
Version Complète : MisterAI/WM-ENT-API-DUMP_FR_2026.07
Nouveau Dépot : https://huggingface.co/datasets/MisterAI/WM-ENT-API-DUMP_FR_2026.07
ESSAI I : Section Introductive Uniquement :: Jeux De Données : Dump WikiMedia Français Juillet 2026 : Extraction et Nettoyage
Description
Ce JDD contient des articles extraits du dump complet de Wikimedia Enterprise de juillet 2026, nettoyés et structurés pour l'entraînement de modèles d'apprentissage… See the full description on the dataset page: https://huggingface.co/datasets/MisterAI/Wiki_FR_2026.07.