Datasetpublic

sapinsapin/halohalo

halohalo Dataset Summary halohalo is a Pretraining text corpus for Philippine languages, assembled from web-scraped data. It is compatible with Fineweb for LLM Pretraining. Source Data Derived from the following cleaned datasets: Source Documents halo-hil 8,874 halo-tgl 6,589 halo-bcl 1,264 Each source dataset was cleaned using clean_halo.py to remove web boilerplate, navigation menus, markdown noise, HTML artifacts, and… See the full description on the dataset page: https://huggingface.co/datasets/sapinsapin/halohalo.

sourceHugging Faceupdated 6mo agoView on Hugging Face

0likes13downloads

Dataset Card

halohalo

Dataset Summary

halohalo is a Pretraining text corpus for Philippine languages, assembled from web-scraped data. It is compatible with Fineweb for LLM Pretraining.

Source Data

Derived from the following cleaned datasets:

Source	Documents
`halo-hil`	8,874
`halo-tgl`	6,589
`halo-bcl`	1,264

Each source dataset was cleaned using clean_halo.py to remove web boilerplate, navigation menus, markdown noise, HTML artifacts, and low-quality documents before being included here.

Processing

1.Cleaning (clean_halo.py) — strips boilerplate, HTML, markdown noise; filters documents with fewer than 30 words or less than 40% Latin characters
2.FineWeb formatting (prep_halohalo.py) — adds source, language, token_count, content_hash; deduplicates against existing documents using MD5 content hashing

Processing code is available at github.com/sapinsapin/halohalo.

Statistics

Metric	Value
Total documents	16,727
Total tokens	19,178,582
Avg tokens per document	1,146.6
Min tokens	30
Max tokens	10,552

Languages

Language	Documents	Word Count
`hil`	8,874	9,332,784
`tgl`	6,589	8,208,749
`bcl`	1,264	1,637,049
Total	16,727	19,178,582

Schema

Field	Type	Description
`text`	`str`	Cleaned document text
`id`	`str`	Unique document identifier
`source`	`str`	Source dataset name
`language`	`str`	ISO 639-3 language code
`token_count`	`int`	Whitespace-tokenized word count
`content_hash`	`str`	MD5 hash of text for deduplication
`url`	`str`	Source URL
`date`	`str`	Crawl date
`dump`	`str`	CommonCrawl dump identifier
`title`	`str`	Page title

Usage

python

from datasets import load_dataset

ds = load_dataset("sapinsapin/halohalo")
print(ds["train"][0])