Modelpublic

benderrodriguez/nemotron35-he-archive-2026-07-04

sourceHugging Faceupdated 3mo agoView on Hugging Face

0likes165downloads

Model Card

Hebrew Nemotron 3.5 ASR streaming fine-tune — full archive (final, 2026-07-05)

Complete archive of the Hebrew fine-tune of nvidia/nemotron-3.5-asr-streaming-0.6b on a vast.ai H100. Repo of record (scripts, RECIPES.md, STATUS.md): git nt-3.5-asr-streaming-ft. Repo was private, made public 2026-07-05 (private-storage quota).

Final model

checkpoints/2026-07-05_03-18-46/nemotron35_he_mix_ct5--val_wer=0.2085-epoch=198.ckpt (= nemo_models/ckpt_mix_final.nemo, weights-only 2.4G; the leg .nemo backup in the same dir is identical). Mixed run: committees 0.5 / CT-v5 0.5 (Lhotse inputcfg), 200k steps total, init from ckpt150k (committees-only run 2). Final global-best valwer 0.20851@199k (CT-v5 dev, canonical Hebrew).

Resume training

Resume point: checkpoints/2026-07-05_03-18-46/...epoch=198-last.ckpt (fp32 + Adam moments + trainer state, 7.7G) via +exp_manager.resume_from_checkpoint=... — see RECIPES.md §4. Training data: committees tars in HF dataset benderrodriguez/knesset-committees-prepped; CT-v5 tars NOT archived (gated source) — re-prep with scripts/prepare_ct5_train.py + scripts_box/prep_ct5_box.sh.

Final eval (canonical WER, streaming attcontext [56,3], targetlang=he-IL)

benchmark	base	com.@20k	@40k	@50k	@150k	mix@49k	@79k	@100k	mix final @199k
CT-v5 test (2000)	0.3570	0.3058	0.2792	0.2671	0.2733	—	0.1947	0.1906	0.1806
committees dev (946)	0.4620	0.3391	0.3196	0.3135	0.2959	—	0.2871	0.2918	0.2826
eval-d1 (17 long-form)	0.1892	0.1911	0.1713	0.1700	0.1543	0.1394	0.1393	0.1367	0.1374
eval-whatsapp (54)	0.3273	—	—	—	0.2658	0.2062	0.2011	0.1994	0.1998

All numbers rescored 2026-07-05 from the archived prediction manifests in outputs/ with scripts/score_canonical.py (identical test subsets throughout; earlier notes mislabeled ckpt50k results as "base").

Scoring: scripts/score_canonical.py; predictions in outputs/eval_*.

val_wer milestones (mix run, CT-v5 dev)

0.2283@37k → 0.22017@53k → 0.21645@73k → 0.2132@104k → 0.21206@128k → 0.21092@133k → 0.20949@139k → 0.20886@194k → 0.20851@199k (final).

Layout

—checkpoints/<run|leg>/ — best + -last Lightning .ckpt per leg (7.7G each)
—nemo_models/ — deployable .nemo milestones (ckpt20k/40k/50k/150k, ckptmix49k/79k/100k, ckptmix_final)
—nemo_experiments_meta/ — full exp tree w/o ckpt/nemo: TensorBoard events, hparams.yaml, cmd-args for every leg
—logs/ — all console logs (train legs incl. crash-preserved, evals, uploads)
—outputs/ — eval prediction manifests for every eval run
—scripts_box/ — box-only scripts not in git
—MANIFEST_all_ckpts_on_box.txt — ls -laR at first-archive time

Crash history (mix run, 9 recoveries)

OOM@61.4k, numba@66.2k, OOM@103.5k (BD 100→70), ENOSPC@106k+107k, numba@131k, OOM@142.1k (BD→60), OOM@197.1k, numba@199k. RNNT grad spikes ~26G; numba = cuLaunchKernel CUDAERRORINVALIDVALUE. Preserved logs: logs/trainmix_*.log.