khaledyusuf44/somaliweb-v1
SomaliWeb v1 — Quality-filtered Somali web corpus 📄 Paper: arXiv:2605.18232 — SomaliWeb v1: A Quality-Filtered Somali Web Corpus with a Matched Tokenizer and a Public Language-Identification Benchmark 💻 Construction pipeline (MIT): github.com/khaledyusuf44/somali-corpus SomaliWeb v1 is a cleaned, deduplicated, and quality-filtered Somali-language web corpus of ~303 million tokens (819,322 documents), built by aggregating three public Somali-heavy web distributions (HPLT v2… See the full description on the dataset page: https://huggingface.co/datasets/khaledyusuf44/somaliweb-v1.
docs: add arXiv:2605.18232 citation, badge, and paper reference
Remove redundant validation.jsonl (replaced by validation.parquet)
Update dataset card YAML for parquet shards
Add train-00001-of-00004.parquet (parquet/zstd)
Add train-00002-of-00004.parquet (parquet/zstd)
Add train-00003-of-00004.parquet (parquet/zstd)
Add train-00000-of-00004.parquet (parquet/zstd)
Add validation.parquet (parquet/zstd)
Add validation split (40,967 docs / ~15M tokens)
Add SHASUMS manifest
Add manual inspection sample (20 docs)
Add HPLT-raw comparison tokenizer
Add SomaliWeb v1 BPE-16K tokenizer
Add dataset card
initial commit