Datasetpublic

BlossomsAI/vietnamese-corpus

Vietnamese Combined Corpus Dataset Statistics Total documents: {<15M:,} Wikipedia articles: {>1.3M:,} News articles: {>13M:,} Text documents: {>200K:,} Processing Details Processed using Apache Spark Minimum document length: {10} characters Text cleaning applied: HTML/special character removal Whitespace normalization URL removal Empty document filtering Data Format Each document has: 'text': The document content 'source':… See the full description on the dataset page: https://huggingface.co/datasets/BlossomsAI/vietnamese-corpus.

sourceHugging Faceapache-2.0updated 2y agoView on Hugging Face

8likes636downloads

Dataset Card

Vietnamese Combined Corpus

Dataset Statistics

—Total documents: {<15M:,}
—Wikipedia articles: {>1.3M:,}
—News articles: {>13M:,}
—Text documents: {>200K:,}

Processing Details

—Processed using Apache Spark
—Minimum document length: {10} characters
—Text cleaning applied:
—HTML/special character removal
—Whitespace normalization
—URL removal
—Empty document filtering

Data Format

Each document has:

—'text': The document content
—'source': Origin of the document (wikipedia/news/text)

Usage Example

from datasets import load_dataset

# Load full dataset
dataset = load_dataset("{username}/{dataset_name}")

# Filter by source
wiki_docs = dataset.filter(lambda x: x["source"] == "wikipedia")

Updates

Released: 2024-12-17