CoolFace
Datasetpublic

tomron87/hebrew-wikipedia-sentences-corpus

Hebrew Wikipedia Sentences Corpus A corpus of 10,999,257 cleaned, deduplicated Hebrew sentences extracted from 366,610 Hebrew Wikipedia articles. Dataset Description This dataset contains Hebrew sentences extracted from Hebrew Wikipedia (crawled 2026-02). Each sentence has been cleaned, filtered for quality, and deduplicated. The dataset is intended for Hebrew NLP tasks including language modeling, text classification, NER, sentence similarity, and more.… See the full description on the dataset page: https://huggingface.co/datasets/tomron87/hebrew-wikipedia-sentences-corpus.

sourceHugging Facecc-by-sa-3.0updated 7mo agoView on Hugging Face
0likes45downloads
Dataset Card

Hebrew Wikipedia Sentences Corpus

A corpus of 10,999,257 cleaned, deduplicated Hebrew sentences extracted from 366,610 Hebrew Wikipedia articles.

Dataset Description

This dataset contains Hebrew sentences extracted from Hebrew Wikipedia (crawled 2026-02). Each sentence has been cleaned, filtered for quality, and deduplicated. The dataset is intended for Hebrew NLP tasks including language modeling, text classification, NER, sentence similarity, and more.

Source

Hebrew Wikipedia via the MediaWiki API.

Processing Pipeline

  1. 1.Crawl — fetched all Hebrew Wikipedia articles via generator=allpages + prop=revisions
  2. 2.Extract — converted wikitext to plain text, split into sentences using rule-based tokenization, filtered by length (5–50 words), Hebrew ratio (≥50%), and content quality
  3. 3.Deduplicate — removed exact duplicate sentences via SHA-256 hashing

Schema

ColumnTypeDescription
sentence_idstringUnique ID (wiki_{article_id}_{sentence_idx})
sentencestringClean Hebrew sentence
article_idint64Wikipedia article page ID
article_titlestringArticle title
categoriesstringPipe-separated Wikipedia categories
sentence_positionint64Position of sentence within the article (0-indexed)
word_countint64Number of whitespace-delimited tokens
hebrew_ratiofloat64Ratio of Hebrew characters to total alphabetic characters

Statistics

MetricValue
Total sentences10,999,257
Unique articles366,610
Word count (mean)16.6
Word count (median)15
Word count (range)5–50
Hebrew ratio (mean)0.982
Hebrew ratio (median)1.000

Usage

python
from datasets import load_dataset

ds = load_dataset("tomron87/hebrew-wikipedia-sentences-corpus")
print(ds["train"][0])

Intended Uses

  • —Hebrew language modeling and pretraining
  • —Text classification and NER
  • —Sentence similarity and semantic search
  • —Hebrew NLP research and benchmarking

Limitations

  • —Register: Wikipedia text is encyclopedic and formal; it does not represent spoken Hebrew, social media, or informal writing.
  • —Temporal: Content reflects Hebrew Wikipedia as of 2026-02. Articles added or modified after this date are not included.
  • —Bias: Wikipedia's coverage is uneven across topics and may reflect systemic biases in editor demographics.

License

CC BY-SA 3.0, inherited from Wikipedia content. See Creative Commons Attribution-ShareAlike 3.0.

Citation

bibtex
@dataset{hebrew_wikipedia_sentences,
  title = {Hebrew Wikipedia Sentences},
  author = {Tom Ron},
  year = {2026},
  url = {https://huggingface.co/datasets/tomron/hebrew-wikipedia-sentences},
  license = {CC BY-SA 3.0},
  note = {Generated on 2026-02-14}
}