zsophia/slm-speech-coach

SLM Speech Coach — audio speech-coaching pairs

License & attribution

This dataset is released under CC-BY-4.0. It is a derivative work that combines:

• —Real speech from The People's Speech (MLCommons), licensed CC-BY-4.0 — please retain this attribution when redistributing.
• —Synthetic child speech generated via TTS (OpenAI gpt-4o-mini-tts / Gemini), plus one controlled delivery flaw injected per clip.
• —Coaching text written by a Gemini 3.1 teacher model (model-generated).

Attribution: "The People's Speech" © MLCommons, CC-BY-4.0. Downstream users must credit the original People's Speech corpus. The Gemma-based models trained on this data are separately governed by the Gemma Terms of Use.

Fine-tuning data for a warm, honest speech coach built on Gemma-4-E4B (audio-native). Each example is a chat conversation: a system prompt, a user turn carrying a degraded practice-speech `.wav` plus a short text question, and an assistant turn with strength / fix / assurance coaching feedback.

Layout

data/gemma_full_{train,val,dev}.jsonl   # conversations (relative audio paths)
audio-data/*.wav                         # the referenced audio (~3.6 GB)

The JSONL is the metadata — no separate metadata.json is needed. Each row has clip_id, an audio list, and messages (with an audio part whose path is relative to the repo root, e.g. audio-data/xxx.wav).

Configs — regenerated_v3 is the current best

`regenerated_v3` (gemma_full_NEW3_*) — a full rebuild, not a patch. 8,827 verified records over 5,940 clips (train 7,062 / val 894 / dev 871). Unlike v1/v2 (one response per clip), v3 is a per-flaw fan-out: each clip yields one coaching response per measured delivery flaw (from verify.detect_flaws), so the model can't map "degradation artifact → one canned answer" — it must ground in the actual delivery. Key changes vs v2:

• —Filler-quotes-only grounding — the teacher quotes ONLY crutch words (um/like); everything else is localized by position ("in your opening") and delivery ("the stretch where your voice went flat"). Kills the content-quote-hallucination incentive at the source (the prompt itself).
• —Negative / honest-when-good examples (~18%) — clean clips get an explicit "nothing major to change," teaching the present/absent boundary (research: +11.2 pts on audio-hallucination).
• —Recalibrated grounding truth — verify.py thresholds re-tuned for children (kids' median WPM 115 sat below the old "slow" cutoff; ~78% of clips were false-flagged on pace) + a transcript-quality guard so the coached flaws are real.

Verified at 99% structural+grounding pass (focus-aware verifier). See DATASET_RESEARCH.md and BRAINLIFT.md in the code repo for the full rationale and behavior thesis.

Splits: train 7,083 / val 879 / dev 865, grouped by true human speaker (--split-by speaker). This fixes a leak present in earlier configs: a kid's person_id embeds a per-recording hash (kid_02_video_games__70a64bb7c8a6 vs ...__b6369fdda672 are the SAME child), so grouping by person_id scattered 4 of 118 kids across train/eval. v3 keys kids by their kid_NN prefix (peoples_speech collapses to source recording), verified 0 speaker leaks across splits — so eval scores aren't inflated by hearing a training voice.

The scripts/verify.py in this repo is the v3 verifier: kid-recalibrated flaw thresholds (kids' median WPM ~115 sat below the old "slow" cutoff, false-flagging ~78% of clips on pace)

• —a focus-aware grounding check (trusts the fan-out focus_dim, which is chosen from detect_flaws, instead of re-deriving it from noisy fix-text regex).

from datasets import load_dataset
ds = load_dataset("zsophia/slm-speech-coach", "regenerated_v3", split="train")

`regenerated_v2` (gemma_full_NEW2_*) — the prior best. Built from regenerated with two eval-driven target fixes (system prompt unchanged, splits/audio/metrics identical):

• —content quotes removed (kept filler quotes) — the audio-only model can't recall words, so quoting content taught it to fabricate (~65% quote-hallucination in the v1 fine-tune).
• —mislabeled timing targets repaired — speed/pause clips whose fix coached the wrong dimension.

Validated on dev before training (LLM-judge on the targets): correct 93.9%, hallucinated-flaw 6.6%, hallucinated-quote 0.0% — vs the v1 fine-tune's output at 49% / 55% / 65%.

Below covers the earlier default vs regenerated distinction:

Configs: default vs regenerated

• —`default` — the original one-shot teacher labels (gemma_full_*.jsonl).
• —`regenerated` — same clips, same splits, same audio (gemma_full_NEW_*.jsonl), with two data-quality passes applied. regenerated is derived from default, not an independent dataset: ~2,050 non-quoted rows are byte-identical, and only the prompt and (for quoting rows) feedback fields differ.

What changed in regenerated:

• —Quoting rows (~1,464): feedback rewritten so any quoted student words are grounded in the transcript (invented "you said …" quotes removed), the reused stock prompts are replaced with a unique tone-matched message, and the overused "let your voice lift" phrasing is de-clichéd.
• —Non-quoting rows (~600): prompt-only refresh — a fresh unique prompt, feedback kept verbatim (those never hallucinated).
• —Remaining non-quoting rows keep the original prompt + feedback.

Prompt uniqueness rises from 38% → 51%; membership is identical to default (verified 0 cross-split leakage, same clip→split map).

from datasets import load_dataset
ds = load_dataset("zsophia/slm-speech-coach", "regenerated", split="train")

Loading

Because audio paths are relative to the repo root, download the whole repo and point the trainer's --audio-root at it:

hf download zsophia/slm-speech-coach --repo-type dataset --local-dir slm-speech-coach
python train_lora_colab.py \
  --train slm-speech-coach/data/gemma_full_train.jsonl \
  --val   slm-speech-coach/data/gemma_full_val.jsonl \
  --audio-root slm-speech-coach \
  --out out/gemma_full_lora

Splits

No leakage. Splits are grouped by source speech, so no degraded variant of a training speech appears in validation or dev. Verified disjoint at both levels:

• —exact clip_id overlap between any two splits: 0
• —base-speech overlap (degradation suffix stripped, e.g. __long_pauses): 0