Haopeng/wavlm-large-s3t2ss-libri960-ce-segmentation

WavLM-Large · S3T2** · FiLM · Libri960 → Libri100 CE

This repository exports the checkpoint used as “Ours 10ms” in the historical LibriSpeech → TIMIT segmentation comparison. It includes the full trained WavLM backbone, projection, FiLM module, recognition heads, labels, and an explicitly frozen encoder adapter for PRiSM representation probing. The decoder addendum includes standalone historical fused/frame_merge inference, the original saved-boundary scorer, and an audio CLI; see DECODING.md.

Selected checkpoint: CKPT+step50_keepall_globalstep000000450_epoch003.ckpt. The final checkpoint is epoch 3 / step 450 of a Libri100 alignment-CE refinement initialized from a Libri960 Subphonetic OTTC curriculum. It is not the later Libri960 recognition-PER-best epoch-17 model, and it is not an OTTC-only final stage.

中文说明：这是此前 segmentation 主表中的 S3T2** + FiLM 版本。训练链包括 Libri960 阶段和后续 Libri100 CE 校准；Libri100 是 Libri960 的子集。上传时保留 完整表示网络，供 PRiSM 冻结编码器、训练新的下游预测头。原始识别 logits 不是 PRiSM 下游探针的默认输入。

What is in the model

“10 ms” describes the interpolated topology stream. The WavLM convolution strides remain [5, 2, 2, 2, 2, 2, 2] (320 input samples); this is not a native 10 ms WavLM convolution conversion. Use the returned lengths, not samples // 160.

Let h be the 384-dimensional projection and p = softmax(content(h)). Both h and p are independently interpolated to twice the frame count. FiLM computes:

gamma, beta = FiLM_network(interpolate(p))
fused = interpolate(h) * (1 + 0.5 * tanh(gamma)) + 0.5 * beta

Computing the content posterior after interpolating h changes this computation. The included implementation preserves the original order.

Training and checkpoint selection

1. 1.Microsoft WavLM-Large self-supervised initialization. Its upstream pretraining is broader than LibriSpeech960; “Libri960” refers to the local supervised chain.
2. 2.Earlier LibriSpeech100 context-phone / subphonetic / CE initialization.
3. 3.LibriSpeech960 S3T2** + FiLM curriculum. The next stage started from epoch 9: CKPT+009_topology_PER_9.5073_MDDF1_0.0000_TSE_51.4ms_MAE_25.7ms_ACC20_0.279.ckpt.
4. 4.Frame-wise alignment CE on clean LibriSpeech100, using the fused topology-phone branch. training_alignment_ce_only=true; the final objective is CE, even though OTTC coefficients remain in the inherited training YAML.
5. 5.The exported epoch-3 / step-450 checkpoint had the lowest TIMIT-dev closure-merged, non-silence forced-alignment TSE in that CE checkpoint sweep. It was subsequently used in the text-independent segmentation comparison.

This is the historically selected segmentation/alignment model, not a claim of the highest segmentation F1 over every historical run. TIMIT test results were also inspected during the earlier curriculum development, so the TIMIT numbers below are historical development results, not an untouched benchmark claim.

provenance.json records the checkpoint identity, original weight-file hashes, the training chain, tensor counts, and export versions. No training recordings, transcripts, optimizer state, or credentials are distributed in this package.

Historical segmentation results

Text-independent fused / frame_merge inference; no transcript is supplied to the segmenter. Scoring uses non-silence internal phone boundaries, 20 ms tolerance, and the directional greedy ss-phoneme-seg recheck performed on 2026-06-22.

Separate transcript-conditioned forced-alignment results, with raw TIMIT PHN references, compatible stop-closure merging, and non-silence scoring:

These are different evaluation protocols. Do not report the forced-alignment TSE as the error of free segmentation. Machine-readable values and protocols are in metrics.json. Full historical scoring has not been rerun on this export.

Load for PRiSM

Install requirements.txt into the environment used by your PRiSM checkout. The loader is explicit Python code rather than an AutoModelForCTC checkpoint:

from pathlib import Path
import sys
from huggingface_hub import snapshot_download

# Replace this with the published repository and immutable commit.
bundle = Path(snapshot_download("Haopeng/wavlm-large-s3t2ss-libri960-ce-segmentation",
                                revision="COMMIT_SHA"))
sys.path.insert(0, str(bundle))
from spottc import load_prism_encoder

encoder = load_prism_encoder(bundle, feature_view="fused", device="cuda")
# speech: float32 [batch, padded_samples], mono audio at 16 kHz.
# speech_lengths: int64 [batch], lengths in SAMPLES, not relative fractions.
representations, frame_lengths = encoder.encode(speech, speech_lengths)
assert representations.shape[-1] == encoder.encoder_output_size() == 384

For a local exported directory, pass that path directly; no network is required. To test a real audio file:

python prism_example.py example.wav --model . --device cuda

PRiSM only requires encode() and encoder_output_size() for the frozen audio encoder. Its AttentionMLPHead receives the returned continuous frames and their valid lengths. Train that new prediction head with the task's normal labels and train/validation/test splits. Do not substitute the 122-dimensional state logits or the 259-dimensional content logits for these 384-dimensional features.

A PRiSM Hydra model.net configuration can be:

_target_: spottc.load_prism_encoder
repo_id_or_path: /absolute/path/to/downloaded/model
feature_view: fused

Replace the whole model.net object, and ensure the downloaded directory is on PYTHONPATH; fields for other PRiSM backbones do not apply to this factory. Set model.freeze_encoder=true. The adapter additionally keeps all encoder modules in eval mode when PRiSM recursively calls .train().

Useful representation ablations:

Keep the view in the experiment name and results. A backbone-only probe does not measure the complete representation that produced the segmentation result. For comparison with ZIPA, the probe architecture is the same but its input and hidden dimensions differ; record head parameter counts and training budgets.

Recognition outputs and preprocessing

from spottc import SPOTTCModel
model = SPOTTCModel.from_pretrained(bundle, device="cuda")
outputs = model(speech, speech_lengths, view="all", normalization="speechbrain")
fused_logits = outputs["fused_logits"]       # [B, topology_frames, 122]
valid_frames = outputs["topology_lengths"]

Use torch.no_grad() or inference mode for direct model inference. The PRiSM adapter already disables gradients and freezes the model.

• —The PRiSM adapter normalizes each unpadded waveform and then pads with an attention mask, matching PRiSM's WavLM preprocessing.
• —The historical SpeechBrain wrapper normalizes the padded waveform with layer_norm and does not supply an attention mask. normalization="speechbrain" exposes that behavior; use batch size 1 for independent-utterance decoding.
• —The two preprocessing paths need not produce identical outputs. The PRiSM probing path is not claimed to reproduce all historical segmentation scores.
• —labels.json contains the exact state and context-label inventories. Labels are ARPAbet-derived English labels, not IPA strings. PRiSM phone recognition requires explicit label conversion and the correct phone decoder; the frozen representation probe does not need that conversion.
• —S3T2** state IDs are not plain CTC phone IDs. Use historical_decoder.py for the extracted frame_merge algorithm and state-to-phone merging. The original saved-boundary scorer is included. Reference normalization and the task's IPA mapping remain explicit steps; see DECODING.md.

Validation and scope

All 512 tensors (488 backbone + 24 task-network tensors) are exported with strict key/shape checks and exact equality to the source checkpoint. Weights use safetensors; no pickle checkpoint loading is required for consumers. The CPU compatibility check passed on two real, short audio clips with unequal lengths. The original PRiSM AttentionMLPHead completed three optimizer steps with finite losses; the head changed while the encoder's parameter hash remained identical. Recursive training mode preserved encoder freezing and eval mode. Both waveform normalization paths produced finite outputs.

validation.json records these checks, and SHA256SUMS covers all package files except the manifest itself. A full PRiSM downstream benchmark has not been completed for this exported checkpoint in this session.

decoder_validation.json separately records comparisons with the original decoder functions, rescoring archived timestamps, and a real-audio CLI smoke test. Rescoring archived outputs does not rerun historical model inference.

The package is a research model for English phonetic representation and boundary analysis. No additional model-weight license is asserted by this export; consult the repository owner and the upstream WavLM terms for redistribution/use terms.