stukenov/sozkz-corpus-dedup-kk-web-v1

sozkz-corpus-dedup-kk-web-v1 Deduplicated Kazakh web text corpus collected from 6 public HuggingFace datasets. Contains only texts not present in kz-transformers/multidomain-kazakh-dataset (12.4M texts were used as the dedup reference). Stats Field Value Total unique texts 9,475,089 Format Parquet (142 shards) Columns text, source Dedup method MD5 hash (exact match) Dedup reference kz-transformers/multidomain-kazakh-dataset (12.4M hashes) Date… See the full description on the dataset page: https://huggingface.co/datasets/stukenov/sozkz-corpus-dedup-kk-web-v1.