trungbb8/vietnamese-news-copus-segmented

Dataset Card for vietnamese-news-copus-segmented

Dataset Summary

This dataset is a refined collection of Vietnamese news articles, originally sourced from ademax/binhvq-news-corpus. It has been processed through a specialized pipeline for cleaning, normalization, and word segmentation. It is ideal for training Vietnamese Language Models (LLMs), word embeddings, or text classification tasks.

• —Original Source: ademax/binhvq-news-corpus
• —Language: Vietnamese (vi)
• —Format: Word-segmented text (using VnCoreNLP)

Data Processing Pipeline

The dataset was constructed using the following automated pipeline:

1. 1.HTML Cleaning: Stripped all HTML tags and boilerplate using BeautifulSoup.
2. 2.Text Normalization:
3. 3.Applied Unicode NFC normalization.
4. 4.Standardized punctuation (quotes, dashes, ellipses).
5. 5.Removed control characters and normalized whitespaces.
6. 6.Boilerplate & Signature Removal:
7. 7.Used Regex to remove common "See more", "Source", and "Photo by" patterns.
8. 8.Applied heuristic rules to detect and strip journalist names, contributor info (CTV), and editorial signatures at the end of articles.
9. 9.Length Filtering: Retained only high-quality articles with character lengths between 500 and 20,000.
10. 10.Deduplication: Performed exact deduplication using MD5 hashing to ensure data uniqueness.
11. 11.Word Segmentation: Segmented compound words using VnCoreNLP (e.g., trí tuệ nhân tạo becomes trí_tuệ nhân_tạo).

Dataset Structure

The dataset contains a single column:

• —text (string): The final cleaned and word-segmented Vietnamese text.

Usage

You can load this dataset using the Hugging Face datasets library:

from datasets import load_dataset

dataset = load_dataset("trungbb8/vietnamese-news-copus-segmented")
print(dataset['train'][0]['text'])

Technical Specifications

• —Segmentation Tool: VnCoreNLP (wseg annotator).

• —Storage: Sharded at max 500MB per file for efficient streaming and downloading.

Limitations

• —Heuristics: While the cleaning process is robust, some unconventional author signatures might remain or very short concluding sentences might be accidentally removed.

• —Segmentation: If your model requires syllable-level input, simply replace underscores (_) with spaces.

License

This dataset is derived from the binhvq-news-corpus. Users should refer to the original source's licensing terms.

Note: This dataset was uploaded using an automated pipeline. For any inquiries, please contact the repository owner.