Datasetpublic

VTSNLP/vietnamese_curated_dataset

Dataset Description Vietnamese Curated Text Dataset. This dataset is collected from multiple open Vietnamese datasets, and curated with NeMo Curator Developed by: Viettel Solutions Language: Vietnamese Details Please visit our Tech Blog post on NVIDIA's plog page for details. Link Data Collection We utilize a combination of datasets that contain samples in Vietnamese language, ensuring a robust and representative text corpus. These datasets… See the full description on the dataset page: https://huggingface.co/datasets/VTSNLP/vietnamese_curated_dataset.

sourceHugging Faceupdated 2y agoView on Hugging Face

76likes1.2kdownloads

Dataset Card

Dataset Description

Vietnamese Curated Text Dataset. This dataset is collected from multiple open Vietnamese datasets, and curated with NeMo Curator

—Developed by: Viettel Solutions
—Language: Vietnamese

Details

Please visit our Tech Blog post on NVIDIA's plog page for details. Link

Data Collection

We utilize a combination of datasets that contain samples in Vietnamese language, ensuring a robust and representative text corpus. These datasets include:

—The Vietnamese subset of the C4 dataset .
—The Vietnamese subset of the OSCAR dataset, version 23.01.
—Wikipedia's Vietnamese articles.
—Binhvq's Vietnamese news corpus.

Preprocessing

We use NeMo Curator to curate the collected data. The data curation pipeline includes these key steps:

1.Unicode Reformatting: Texts are standardized into a consistent Unicode format to avoid encoding issues.
2.Exact Deduplication: Removes exact duplicates to reduce redundancy.
3.Quality Filtering:
4.Heuristic Filtering: Applies rules-based filters to remove low-quality content.
5.Classifier-Based Filtering: Uses machine learning to classify and filter documents based on quality.

[Notebook](https://github.com/NVIDIA/NeMo-Curator/blob/main/tutorials/pretraining-vietnamese-data-curation/pretraining-vietnamese-data-curation.ipynb)

Dataset Statistics

Content diversity <img src="https://cdn-uploads.huggingface.co/production/uploads/661766c00c68b375f3f0ccc3/mW6Pct3uyP_XDdGmE8EP3.png" alt="Domain proportion in curated dataset" width="500"/>

Character based metrics <img src="https://cdn-uploads.huggingface.co/production/uploads/661766c00c68b375f3f0ccc3/W9TQjM2vcC7uXozyERHSQ.png" alt="Box plots of percentage of symbols, numbers, and whitespace characters compared to the total characters, word counts and average word lengths" width="900"/>

Token count distribution <img src="https://cdn-uploads.huggingface.co/production/uploads/661766c00c68b375f3f0ccc3/PDelYpBI0DefSmQgFONgE.png" alt="Distribution of document sizes (in terms of token count)" width="500"/>

Embedding visualization <img src="https://cdn-uploads.huggingface.co/production/uploads/661766c00c68b375f3f0ccc3/sfeoZWuQ7DcSpbmUOJ12r.png" alt="UMAP visualization of 5% of the dataset" width="650"/> UMAP visualization of 5% of the dataset