sapinsapin/pld

Philippine Language Dataset (PLD)

Ten Philippine languages, 980 speakers, 448 hours of prompted speech — one of the largest multilingual Philippine speech collections available as Parquet.

<div align="center">

334,268 utterances · 448.2 hours · 980 speakers · 10 languages · 16kHz mono

 

[▶ Try the models in your browser](https://huggingface.co/spaces/sapinsapin/halohalo-dashboard) — transcribe, synthesize, or convert a voice in any of the ten languages, from your microphone or the preloaded clips.

</div>

Collected by the University of the Philippines Diliman Digital Signal Processing Laboratory. Every row is one prompted recording: the corpus ships pre-segmented WAVs with the prompt text stored inline in each session log, so no forced alignment or segmentation was applied here.

Most Philippine language speech data stops at Tagalog. This one covers Bikol, Cebuano, Kapampangan, Hiligaynon, Ilocano, Waray, Pangasinan and Tausug at scale — languages with tens of millions of speakers and almost no public ASR/TTS data.

Read [`text_is_prompt`](#-read-this-before-training) before you train. 2,586 rows carry an elicitation question instead of a transcript, and silently training on them will poison your model.

30-second quickstart

from datasets import load_dataset

ds = load_dataset("sapinsapin/pld", split="train", streaming=True)
row = next(iter(ds))

print(row["language_name"], "|", row["sentence"])
print(row["speech_type"], row["duration"], "s")

The training-ready filter, in full:

ds = load_dataset("sapinsapin/pld", split="train")
ds = ds.filter(lambda x: not x["text_is_prompt"] and 0.3 <= x["duration"] <= 30.0)

One language at a time:

bikol = ds.filter(lambda x: x["language"] == "bcl")

⚠️ Read this before training

text_is_prompt — 2,586 rows have no transcript

Rows where speech_type == "spontaneous" do not carry a transcript. The session logs store the elicitation question that was put to the speaker — e.g. "Saen an dream destination mo?" — while the audio is 20–90 seconds of their free-speech answer. The same question text repeats verbatim across different speakers.

They are kept because the audio is genuine spontaneous speech (32 hours of it, valuable for pretraining, VAD, diarization, or re-transcription), but they are poison for supervised (audio, text) training:

ds = ds.filter(lambda x: not x["text_is_prompt"])

Other things to know

What's inside

Languages

English word and sentence lists (EngW.txt, EngSen.txt) were read by the same speakers. Those rows are labeled language = "eng", while corpus_language retains the Philippine language collection they came from, so per-language filters stay clean either way.

Speech types

Splits

Schema

The dialect fields are unusually rich for a speech corpus — speaker, mother and father dialect are all recorded, which supports dialectometry and language-contact work that most corpora can't.

Models trained on this data

Eleven reference finetunes — a TTS model per language plus one multilingual speech-to-speech model — each with listen-test samples in its samples/ directory. All were trained on a single 8 GB GPU with `finetune_tts.py` / `finetune_s2s.py`, so they are baselines to hear and beat, not state-of-the-art:

Speech-to-speech: `speecht5_vc-pld` — any-to-any voice conversion across all ten languages, trained on same-sentence cross-speaker pairs mined from PLD's shared prompt lists (the corpus has no parallel translations, but many speakers reading the same prompt is exactly the parallel data voice conversion needs).

Speech recognition — one model per language

openai/whisper-small finetuned per language, scored on a held-out split. CER is the selection metric, not WER: Philippine-language orthography varies at the word level (hyphenation and affix spelling especially), which punishes WER for differences a reader would not consider errors.

Read these as in-domain numbers: PLD is prompted read speech recorded in controlled sessions, and the split is random over utterances, so speakers overlap between train and test. Expect materially worse performance on spontaneous or noisy audio, and re-split on speaker_id if you need a speaker-disjoint measurement.

Two caveats worth stating plainly. Whisper's decoder only has language tokens for about 100 languages: English and Tagalog are in the vocabulary, the other eight here are not, so they train under the closest token (<|tl|>) which finetuning repurposes as the language slot. And Pangasinan is data-limited (~5.5k utterances total in the corpus), which is the main reason it trails.

Reproduce any of them in one command:

python finetune_tts.py --dataset pld --language ceb --push
python finetune_asr.py --dataset pld --language ceb --push
python finetune_s2s.py --push

If you train something better on PLD, tag this dataset in your model card and it will appear here.

How it was built

1. 1.Walk every session directory; parse the per-session .log (speaker demographics header, then one row per utterance: WAV name, prompt list, prompt text).
2. 2.Classify each utterance's speech_type. Explicit markers (_Iso_, _Utt_, Spontaneous, digits) are used where present; the corpus uses at least five naming conventions, so the ~55k rows with no marker are typed by the measured median word count of their prompt list rather than by guessing from the filename.
3. 3.Repair double-encoded UTF-8 in transcripts (hapúnan → hapúnan) — 150 of 166 affected lines recover; the rest are left intact rather than risk a worse string.
4. 4.Resample to 16 kHz mono, encode FLAC, shard to Parquet, 90/10 random split.

1,943 rows (0.6%) reference WAVs that are not present in the archive and were skipped.

Pipeline source: `process_pld_parquet.py` · parser: `halolib/pld.py`

Limitations

• —Prompted, not conversational. Except for the 32 h spontaneous portion, this is people reading from lists. Prosody and vocabulary reflect that.
• —Transcripts are unverified prompts. Nobody checked that speakers read the prompt exactly; expect a residual mismatch rate.
• —Coverage is uneven — Bikol has 95.8 h, Tausug 5.9 h. Don't expect balanced multilingual behaviour without resampling.
• —Recording conditions vary by session and are only loosely described in session_environment.
• —No held-out speaker split is provided by default.
• —Language codes follow ISO 639-3; fil and tgl distinctions in the wild are inconsistent, so filter on both if you merge with other corpora.

Related datasets

Part of the halohalo Philippine-language speech family:

License, source and citation

Collected by the UP Diliman Digital Signal Processing Laboratory. This is a repackaging for research use; the underlying corpus terms are those of UP-DSP. Please credit the original collectors, and contact UP-DSP for terms covering uses beyond research.

If you represent UP-DSP and want attribution, terms, or access changed, please open a discussion on this repo.

Contributing

Eight of these ten languages have essentially no public ASR or TTS baseline. That is the opportunity here.

• —Train a baseline on any single language and tag this dataset in your model card
• —Report bad rows via the Community tab (include source_file and session_id)
• —Improve the pipeline: github.com/sapinsapin/halohalo