datasets
Training and evaluation data, with the modality, task and licence stated up front. Listed live from the Hugging Face Hub.
wikipedia-ja-20231030
Wikipedia Japanese data (20231030)
Source Date: 2023/10/30
Source: https://dumps.wikimedia.org/other/cirrussearch/
License
CC BY-SA 4.0
Example
WIP
osm-polygon-wikidata-only
OSM Polygon Wikidata, Wikipedia and Wikivoyage
OSM polygons carrying wikidata=*, enriched with multilingual Wikipedia and Wikivoyage documents. The published tables preserve regional records and provenance.
Source code: GitHub repository.
Dataset snapshot
Metric
Value
Polygon rows across regional extracts
1,184,110
Unique polygon identities (osm_type, osm_id)
1,157,841
Polygons with successful non-empty text (unique OSM identities)
650,663… See the full description on the dataset page: https://huggingface.co/datasets/NoeFlandre/osm-polygon-wikidata-only.wikipedia-paragraphs
wikipedia-paragraphs
wikipedia-paragraphs is a dataset generated from Wikipedia, designed for natural language processing (NLP) research.
Each entry contains cleaned paragraph text and Wikilink information extracted from a Wikipedia page, along with useful metadata such as categories, templates, and the associated Wikidata QID.
Dataset structure
Configurations
The dataset is organized into multiple configurations, such as enwiki-20260607-v1.2.1.… See the full description on the dataset page: https://huggingface.co/datasets/singletongue/wikipedia-paragraphs.wikidata-extraction
Wikidata Extraction
This dataset contains all RDF triples extracted from the latest Wikidata, converted from the N-Triples format to Parquet.
The data originates from Wikidata, a free and open knowledge base that acts as central storage for structured data used by Wikipedia and other Wikimedia projects. The source file is the "truthy" N-Triples dump (latest-truthy.nt.bz2), which contains only the current, non-deprecated statements.
The code to extract this data is available at… See the full description on the dataset page: https://huggingface.co/datasets/piebro/wikidata-extraction.osm-polygon-wikidata-and-wikipedia
OSM Polygon Wikidata + Wikipedia, V2
V2 builds on the V1 Wikidata-only dataset with multilingual Wikipedia and Wikivoyage text. It also retains valid multilingual wikipedia=* references, including polygons without a Wikidata QID.
Source code: GitHub repository.
Dataset snapshot
Metric
Value
Polygon rows across regional extracts
1,259,424
Unique polygon identities (osm_type, osm_id)
1,188,854
Polygons with successful non-empty text (unique OSM… See the full description on the dataset page: https://huggingface.co/datasets/NoeFlandre/osm-polygon-wikidata-and-wikipedia.wikipedia-22-12-en-embeddings-all-MiniLM-L6-v2
Dataset Card for "wikipedia-22-12-en-embeddings-all-MiniLM-L6-v2"
More Information needed
wiki_snippets
Dataset Card for "wiki_snippets"
Dataset Summary
Wikipedia version split into plain text snippets for dense semantic indexing.
Supported Tasks and Leaderboards
More Information Needed
Languages
More Information Needed
Dataset Structure
We show detailed information for 2 configurations of the dataset (with 100 snippet passage length and 0 overlap) in
English:
wiki40b_en_100_0: Wiki-40B
wikipedia_en_100_0: Wikipedia
Data Instances… See the full description on the dataset page: https://huggingface.co/datasets/community-datasets/wiki_snippets.cohere-wiki-sbert
Dataset Card for "cohere-wiki-sbert"
More Information needed
wikipedia_en
wikipedia_en
This is a curated Wikipedia English dataset for use with the II-Commons project.
Dataset Details
Dataset Description
This dataset comprises a curated Wikipedia English pages. Data sourced directly from the official English Wikipedia database dump. We extract the pages, chunk them into smaller pieces, and embed them using Snowflake/snowflake-arctic-embed-m-v2.0. All vector embeddings are 16-bit half-precision vectors optimized for cosine indexing… See the full description on the dataset page: https://huggingface.co/datasets/Intelligent-Internet/wikipedia_en.bhasha-wiki
Dataset Card for Bhasha-Wiki
Translated wikipedia articles
Dataset Details
Dataset is being updated
Dataset Description
We have translated 6.4 million English wikipedia articles into 6 Indic languages. The translations were done using IndicTrans2 model.
Curated by: Soket AI labs
Language(s) (NLP): Hindi, Bengali, Gujarati, Tamil, Kannada, Urdu
License: cc-by-sa-3.0
Uses
For pretraining or Fine tuning for Indic language models
Dataset… See the full description on the dataset page: https://huggingface.co/datasets/soketlabs/bhasha-wiki.wikipedia-utils
Wikipedia-Utils: Preprocessed Wikipedia Texts for NLP
Preprocessed Wikipedia texts generated with the scripts in singletongue/wikipedia-utils repo.
For detailed information on how the texts are processed, please refer to the repo.
dolma_20bn_wiki_upsamplewikimedia-pageview-timeseries-raw
Wikimedia Pageview Time Series — full raw (wide format)
Full, unsampled Wikipedia pageview time series for every Wikimedia
project (Wikipedia, Wiktionary, Commons, etc.), stored as raw wide
parquet files: one row per article, one column per timestamp.
This is the complete derived output of the upstream pipeline —
the companion repo
jeremycochoy/wikimedia-pageview-timeseries
holds a sampled, reshaped version (3.7 M rows in HF long format
for training). Use this repo if you need the… See the full description on the dataset page: https://huggingface.co/datasets/jeremycochoy/wikimedia-pageview-timeseries-raw.phantom-wiki-v0-5-0-predictions
Dataset Card for Dataset Name
Predictions from https://huggingface.co/datasets/mlcore/phantom-wiki-v050
Dataset Details
Dataset Description
Curated by: [More Information Needed]
Funded by [optional]: [More Information Needed]
Shared by [optional]: [More Information Needed]
Language(s) (NLP): [More Information Needed]
License: [More Information Needed]
Dataset Sources [optional]
Repository: [More Information Needed]
Paper [optional]: [More… See the full description on the dataset page: https://huggingface.co/datasets/kilian-group/phantom-wiki-v0-5-0-predictions.LDS-retrain-bank-adamw-wikitext2-N4656-bs8-seed1007
Retrain bank: WikiText-2 / GPT-2, random halves, seed 1007
This repository contains 100 fully retrained language models, not just scores.
Each model is GPT-2 (gpt2) fine-tuned on a different random 50% (2,328 documents)
of the 4,656-document WikiText-2 training set from
EleutherAI/bergson-wikitext-2-4656-chunks,
following the recipe of Bae et al. 2024, Training Data Attribution via Approximate
Unrolled Differentiation (App. B.1). retrained/base is trained on the full set with… See the full description on the dataset page: https://huggingface.co/datasets/EleutherAI/LDS-retrain-bank-adamw-wikitext2-N4656-bs8-seed1007.Wiki-JA-Pair
Wiki-JA-Pair
Wiki-JA-Pair is a dataset of 1M image–text pairs sourced from the Japanese Wikipedia (Wiki-JA).
This dataset is constructed using the May 1, 2025 dump (jawiki-20250501-pages-articles-multistream.xml.bz2).
How to Use
from datasets import load_dataset
ds = load_dataset("llm-jp/Wiki-JA-Pair")
Format
Wiki-JA-Pair includes the following columns:
url: URL of the image
caption: Caption associated with the image
description: Nearby text that… See the full description on the dataset page: https://huggingface.co/datasets/llm-jp/Wiki-JA-Pair.wikianc
Dataset Card for WikiAnc
Dataset Summary
The WikiAnc dataset is an automatically generated dataset from Wikipedia (all languages) and Wikidata dumps (August, 2023).
The code for generating the dataset can be found here.
Supported Tasks
wikificiation: The dataset can be used to train a model for Wikification.
named-entity-linking: The dataset can be used to train a model for Named Entity Linking.
Languages
The text in the dataset is… See the full description on the dataset page: https://huggingface.co/datasets/cyanic-selkie/wikianc.LDS-retrain-bank-adamw-wikitext2-N4656-bs8-seed1004
Retrain bank: WikiText-2 / GPT-2, random halves, seed 1004
This repository contains 100 fully retrained language models, not just scores.
Each model is GPT-2 (gpt2) fine-tuned on a different random 50% (2,328 documents)
of the 4,656-document WikiText-2 training set from
EleutherAI/bergson-wikitext-2-4656-chunks,
following the recipe of Bae et al. 2024, Training Data Attribution via Approximate
Unrolled Differentiation (App. B.1). retrained/base is trained on the full set with… See the full description on the dataset page: https://huggingface.co/datasets/EleutherAI/LDS-retrain-bank-adamw-wikitext2-N4656-bs8-seed1004.wikipedia-pageviews
Wikipedia Article Pageviews
This repository automatically fetches and aggregates the 100 most popular Wikipedia articles by pageviews - creating a dataset that enables tracking trending topics on Wikipedia.
It works by polling the WikiMedia API on a daily basis and fetching the top 100 most popular articles from two days ago.
The fetcher runs in a scheduled GitHub Actions workflow, which is available here.
The dataset begins in the year 2016 and the textual data is presented as… See the full description on the dataset page: https://huggingface.co/datasets/vtasca/wikipedia-pageviews.LDS-retrain-bank-adamw-wikitext2-N4656-bs8-seed1006
Retrain bank: WikiText-2 / GPT-2, random halves, seed 1006
This repository contains 100 fully retrained language models, not just scores.
Each model is GPT-2 (gpt2) fine-tuned on a different random 50% (2,328 documents)
of the 4,656-document WikiText-2 training set from
EleutherAI/bergson-wikitext-2-4656-chunks,
following the recipe of Bae et al. 2024, Training Data Attribution via Approximate
Unrolled Differentiation (App. B.1). retrained/base is trained on the full set with… See the full description on the dataset page: https://huggingface.co/datasets/EleutherAI/LDS-retrain-bank-adamw-wikitext2-N4656-bs8-seed1006.LDS-retrain-bank-adamw-wikitext2-N4656-bs8-seed1008
Retrain bank: WikiText-2 / GPT-2, random halves, seed 1008
This repository contains 100 fully retrained language models, not just scores.
Each model is GPT-2 (gpt2) fine-tuned on a different random 50% (2,328 documents)
of the 4,656-document WikiText-2 training set from
EleutherAI/bergson-wikitext-2-4656-chunks,
following the recipe of Bae et al. 2024, Training Data Attribution via Approximate
Unrolled Differentiation (App. B.1). retrained/base is trained on the full set with… See the full description on the dataset page: https://huggingface.co/datasets/EleutherAI/LDS-retrain-bank-adamw-wikitext2-N4656-bs8-seed1008.wikipedia-22-12-concat-split
Dataset Card for "wikipedia-22-12-concat-split"
More Information needed
wikimedia-commons-documents-ml_beirThis is a copy of https://huggingface.co/datasets/jinaai/wikimedia-commons-documents-ml reformatted into the BEIR format. For any further information like license, please refer to the original dataset.
Disclaimer
This dataset may contain publicly available images or text data. All data is provided for research and educational purposes only. If you are the rights holder of any content and have concerns regarding intellectual property or copyright, please contact us at "support-data… See the full description on the dataset page: https://huggingface.co/datasets/jinaai/wikimedia-commons-documents-ml_beir.zh-tw-wikipedia
台灣正體中文維基百科 (zh-tw Wikipedia)
截至 2023 年 5 月，中文維基百科 2,533,212 篇條目的台灣正體文字內容。每篇條目為一列 (row)，包含 HTML 以及 Markdown 兩種格式。
A nearly-complete collection of 2,533,212 Traditional Chinese (zh-tw) Wikipedia pages, gathered between May 1, 2023, and May 7, 2023. Includes both the original HTML format and an auto-converted Markdown version, which has been processed using vinta/pangu.py.
於 2023 年 5 月 1 日至 5 月 7 日間取自維基百科 action=query & prop=extracts API，內容皆與維基百科網站之台灣正體版本一致，沒有繁簡體混雜的問題。
For development… See the full description on the dataset page: https://huggingface.co/datasets/zetavg/zh-tw-wikipedia.wikipedia_fr_2022French part of Wikipedia embeded with Solon-embeddings-large-0.1.
wiki-talks
Wiki-Talks
The Wiki-Talks dataset is a collection of conversational threads extracted from the talk pages on Wikipedia.
This dataset captures collaborative dialogue, discussion patterns, and consensus-building among Wikipedia contributors.
It is useful for NLP research focused on dialogue, sentiment analysis, and community dynamics.
Details
Currently due to PyArrow incompatibility to the long recursive structures in the dataset there is an intrinsic incompatibility… See the full description on the dataset page: https://huggingface.co/datasets/lflage/wiki-talks.cohere-wiki-embedding-100k
Dataset Card for "cohere-wiki-embedding-100k"
More Information needed
wikipedia_2017
Dataset Card for Dataset Name
This is a Wikipedia dataset correct to "31-12-2017".
Dataset Details
Dataset Description
WikiMedia routinely publishes dumps of Wikipedia, each containing the revision history of articles. We first defined the relevant revision before extracting the article information. Specifically, we select the most recent revision as of December 31st for each year. Consequently, some revisions in our datasets date back several years from the… See the full description on the dataset page: https://huggingface.co/datasets/Ti-Ma/wikipedia_2017.Wikipedia-EN-FA-Accessibility-Bridge
Wikipedia EN-FA Accessibility Bridge
Current, attributable English and Persian Wikipedia article snapshots for pages
created during a 69-day recency window, plus an EN↔FA
counterpart index and static accessibility signals.
The reproducible full baseline is the official 2026-08-01 Wikimedia dump:
6,289,549 English articles without Persian, 129,821 Persian
articles without English, and 22,277,907 namespace-0 pages in the
combined parity index. Redirects are retained in the parity… See the full description on the dataset page: https://huggingface.co/datasets/Reza2kn/Wikipedia-EN-FA-Accessibility-Bridge.wikipedia_2018
Dataset Card for Dataset Name
This is a Wikipedia dataset correct to "31-12-2018".
Dataset Details
Dataset Description
WikiMedia routinely publishes dumps of Wikipedia, each containing the revision history of articles. We first defined the relevant revision before extracting the article information. Specifically, we select the most recent revision as of December 31st for each year. Consequently, some revisions in our datasets date back several years from the… See the full description on the dataset page: https://huggingface.co/datasets/Ti-Ma/wikipedia_2018.