actableai/zipformer-rnnt-v3

zipformer-rnnt-v3

Vietnamese Zipformer2 transducer (RNNT) fine-tuned for Actable meeting audio, starting from `actableai/zipformer-rnnt-v2` and trained on `actableai/vi-meeting-soniox-wer16` with Soniox transcripts as golden labels.

Performance

Meeting dev (Soniox references)

Eval set: session_soniox_dev.jsonl → 5,453 utterances, ~13.8 h Decoding: greedy search, icefall greedy_search_batch Text norm for WER: lowercase, strip punctuation / bracket tags, collapse whitespace Checkpoint: best-valid-loss.pt using `model_avg` weights

Measured 2026-07-27 with eval_zipformer_meeting_wer16.py (full meeting dev).

Validation RNNT loss (during fine-tune)

Loss on meeting dev CutSet at the start of each epoch (icefall validation):

• —Best valid loss: 0.3362 @ epoch 10
• —Best train loss: 0.2452 @ epoch 10
• —Wall time: ~1 h 38 m (single NVIDIA A100 40GB)

Training details

Data

• —Filter: keep train segments where WER(Soniox ref, baseline Qwen hyp) ≤ 16%
• —Labels: Soniox field text only — never Qwen hypotheses
• —Audio: 16 kHz mono session WAVs (full_session.wav + start/end offsets)
• —Mix / replay: none (use_mux=0) — pure meeting wer16
• —Utt duration filter: 0.3–30 s

Optimization

Architecture (unchanged from v2)

Files

Usage (icefall)

import torch, sys
sys.path.insert(0, "/path/to/icefall")
sys.path.insert(0, "/path/to/zipformer_work")  # finetune.py / model defs

from finetune import get_model, get_params, add_model_arguments
from beam_search import greedy_search_batch
import argparse, sentencepiece as spm

parser = argparse.ArgumentParser()
add_model_arguments(parser)
params = get_params()
params.update(vars(parser.parse_args([])))
params.encoder_dim = "192,256,384,512,384,256"
params.num_encoder_layers = "2,2,3,4,3,2"
params.downsampling_factor = "1,2,4,8,4,2"
params.feedforward_dim = "512,768,1024,1536,1024,768"
params.num_heads = "4,4,4,8,4,4"
params.cnn_module_kernel = "31,31,15,15,15,31"
params.encoder_unmasked_dim = "192,192,256,256,256,192"
params.decoder_dim = 512
params.joiner_dim = 512
params.causal = False
params.vocab_size = 3000
params.blank_id = 0
params.context_size = 2
params.use_transducer = True
params.use_ctc = False

model = get_model(params)
ckpt = torch.load("best-valid-loss.pt", map_location="cpu")
state = ckpt.get("model_avg") or ckpt["model"]
model.load_state_dict(state, strict=False)
model.eval().cuda()

sp = spm.SentencePieceProcessor()
sp.load("bpe.model")

# feature: (1, T, 80) float32 fbank; feature_lens: (1,) int
with torch.no_grad():
    encoder_out, encoder_out_lens = model.forward_encoder(feature, feature_lens)
    token_ids = greedy_search_batch(model, encoder_out, encoder_out_lens)
text = sp.decode(token_ids[0])
print(text)

Notes / limitations

• —Optimized for Vietnamese meeting / conversational audio; not re-evaluated on open-domain policy or YouTube suites in this card.
• —Greedy decoding only in the reported WER; beam search may improve further.
• —Transcripts are Soniox pseudo-labels (filtered), not human gold.
• —CTC head from v2 was not fine-tuned / not present in the shipped transducer-only graph.

Changelog