Sherlock-Comms/wikipedia-en-2026-07-01-passages
English Wikipedia Passages, Chunked (2026-07-01) Every English Wikipedia article split into retrieval-sized passages with title and section attached. A clean, dated corpus for RAG — embed it yourself, or use the ready-made vectors and indexes in the companion repos: embeddings · faiss. Contents 17,473,199 passages, 21 GB, JSON Lines (one passage per line). Fields: id (<pageid>#<n>), title, section, text. Line order matches ids.txt / vector row order in the… See the full description on the dataset page: https://huggingface.co/datasets/Sherlock-Comms/wikipedia-en-2026-07-01-passages.
English Wikipedia Passages, Chunked (2026-07-01)
Every English Wikipedia article split into retrieval-sized passages with title and section attached. A clean, dated corpus for RAG — embed it yourself, or use the ready-made vectors and indexes in the companion repos: embeddings · faiss.
Contents
- 17,473,199 passages, 21 GB, JSON Lines (one passage per line).
- Fields:
id(<pageid>#<n>),title,section,text. - Line order matches
ids.txt/ vector row order in the companion datasets.
How it was built
- Source: English Wikipedia dump 2026-07-01, main-namespace, non-redirect.
- Wikitext stripped to prose; split by top-level (level-2) section, keeping the heading as
section(lead section has an emptysection). - Chunked to ~500 tokens (2000 chars) with a one-paragraph overlap; chunks under 200 chars dropped.
- Navigation and citation sections removed (References, External links, See also, Further reading, Notes, Bibliography, and similar).
Known limitation
Body prose only. Infobox and table content is not included, so structured facts (specific dates, counts, awards, specs) are under-represented — the main recall gap measured on factoid benchmarks. A future revision may add them.
Licence
English Wikipedia text, © its authors, released under CC BY-SA 4.0. This chunked derivative is under the same licence; attribute Wikipedia and this dataset on reuse.