rinnieyoung/sea-javanese-cleaned-parquet-v1

SEA Javanese Cleaned Parquet v1

Dataset Summary

This dataset is a cleaned Javanese pretraining corpus exported in Hugging Face parquet format.

Current public sources used in this release:

• —HuggingFaceFW/fineweb-2 / jav_Latn
• —allenai/c4 / jv
• —afrizalha/Centhini-1-Javanese

Cleaning and Deduplication

Current pipeline:

• —basic text cleaning
• —short-text filtering
• —repetition filtering
• —rule-based noise filtering
• —document-level exact deduplication across all included sources

Current deduplication is exact dedup only. Near-duplicate deduplication is not yet included in this release.

Data Files

• —data/train/*.parquet: cleaned merged training corpus
• —metadata/source_metrics.csv: source-level filtering summary
• —metadata/source_inventory.csv: source inventory used in this run

Run Summary

• —language: Javanese
• —sources_total: 3
• —sources_ok: 3
• —rows_seen: 1295664
• —rows_kept: 734489
• —rows_deduped: 1782
• —overallkeeprate: 0.566882
• —dedupdroprate: 0.001375

Evaluation Metrics

Data-level metrics

The current pipeline reports the following source-level and run-level metrics:

• —rows_seen: total number of input records scanned
• —rows_kept: number of records retained after cleaning and exact dedup
• —rows_deduped: number of records removed by exact dedup
• —keep_rate: rows_kept / rows_seen
• —avg_clean_chars: average cleaned document length in characters
• —avg_clean_words: average cleaned document length in word-like tokens
• —too_short_rate: fraction rejected for being too short
• —repetitive_rate: fraction rejected for high token or character repetition
• —paper_rule_reject_rate: fraction rejected by the rule-based quality filters
• —menu_spam_rate: fraction flagged as menu / navigation / spam-like text

These metrics are included in:

• —metadata/source_metrics.csv

Deduplication metrics

The current baseline deduplication setup is:

• —document-level exact dedup
• —applied after text cleaning / normalization
• —applied globally across all included sources in the same run

Reported dedup metric:

• —exact_duplicate_drop_rate = rows_deduped / rows_seen

Current limitation:

• —near-duplicate deduplication is not yet implemented
• —MinHash / LSH / fuzzy duplicate removal is not included in this release

Manual QC metrics

For manual inspection, the pipeline exports:

• —qc/raw_samples/*.jsonl
• —qc/kept_samples/*.jsonl

Recommended human labels for later annotation:

• —usable natural text
• —template / menu / navigation page
• —spam / advertisement / sales copy
• —non-target language
• —malformed / noisy / unreadable text

Recommended manual QC summary metrics:

• —usable_rate
• —spam_or_template_rate
• —non_target_language_rate
• —noise_rate

Training-facing metrics for the next phase

These are not computed automatically in this release, but are intended for the next phase of evaluation:

• —held-out validation perplexity
• —token count after cleaning
• —tokenizer coverage / unknown rate
• —downstream task score if a suitable benchmark exists

Notes

This is a baseline release intended for corpus construction, source comparison, and cleaning evaluation.