CoolFace
Datasetpublic

SultanR/VieMix

VieMix (https://arxiv.org/abs/2512.18834) is a Vietnamese pretraining corpus built by combining six publicly available Vietnamese datasets, applying Vietnamese-specific quality filtering, and performing cross-dataset deduplication. Subsets Subset Description quality_filtered Quality-filtered data before deduplication minhash_deduped Document-level MinHash deduplication matched Documents appearing in 2+ source datasets The matched subset uses… See the full description on the dataset page: https://huggingface.co/datasets/SultanR/VieMix.

sourceHugging Faceotherupdated 1mo agoView on Hugging Face
0likes4kdownloads
Dataset Card

<p align="center"> <a href="https://huggingface.co/collections/AdaMLLab/mixminmatch"> <img src="https://img.shields.io/badge/🤗_Collection-MixMinMatch-blue" alt="MixMinMatch Collection"> </a> </p>

VieMix (https://arxiv.org/abs/2512.18834) is a Vietnamese pretraining corpus built by combining six publicly available Vietnamese datasets, applying Vietnamese-specific quality filtering, and performing cross-dataset deduplication.

Subsets

SubsetDescription
quality_filteredQuality-filtered data before deduplication
minhash_dedupedDocument-level MinHash deduplication
matchedDocuments appearing in 2+ source datasets

The matched subset uses cross-dataset agreement as a signal for quality.

Usage

python
from datasets import load_dataset

ds = load_dataset("AdaMLLab/VieMix", "minhash_deduped")
ds = load_dataset("AdaMLLab/VieMix", "quality_filtered")
ds = load_dataset("AdaMLLab/VieMix", "matched")

Sources

  • —FineWeb-2 (HuggingFaceFW/fineweb-2, vie_Latn)
  • —HPLT 2.0 (HPLT/HPLT2.0_cleaned, vie_Latn)
  • —CulturaX (uonlp/CulturaX, vi)
  • —C4 (allenai/c4, vi)
  • —FinePDFs (HuggingFaceFW/finepdfs, vie_Latn)
  • —SEA-LION Pile v2 (aisingapore/SEA-PILE-v2, vi)

Pipeline

  1. 1.Quality filtering with Vietnamese-specific thresholds (Latin script ratio with diacritics, repetition patterns, line quality)
  2. 2.Document-level MinHash deduplication (5-gram shingles, 14 bands, 8 hashes per band, similarity threshold 0.8)
  3. 3.Cross-source matching to identify documents appearing in 2+ independent sources

Citation

bib
@misc{alrashed2025mixminmatch,
      title={Mix, MinHash, and Match: Cross-Source Agreement for Multilingual Pretraining Datasets},
      author={Sultan Alrashed and Francesco Orabona},
      year={2025},
      eprint={2512.18834v2},
      archivePrefix={arXiv},
      primaryClass={cs.CL},
      url={https://arxiv.org/abs/2512.18834v2},
}

License

See individual source dataset licenses.