Modelpublic

Nampfiev1995/nemotron35-encoder-only-vi-asr

sourceHugging Faceapache-2.0updated 2d agoView on Hugging Face

0likes108downloads

Model Card

Nemotron 3.5 ASR Vietnamese/English Fine-Tune

This folder is a complete native Hugging Face Transformers RNNT model, including the fine-tuned acoustic encoder, RNNT prediction network, joint network, tokenizer, and processor. It is a conversion of a NeMo fine-tuned nvidia/nemotron-3.5-asr-streaming-0.6b checkpoint.

The original checkpoint was trained in NeMo and exported from:

text

FastConformer-Transducer-BPE-Prompt-Streaming.nemo

It can be loaded with AutoProcessor and AutoModelForRNNT.

Usage

python

import torch
import soundfile as sf
from transformers import AutoModelForRNNT, AutoProcessor

MODEL_ID = "Nampfiev1995/nemotron35-encoder-only-vi-asr"

processor = AutoProcessor.from_pretrained(MODEL_ID, trust_remote_code=True)
processor.set_num_lookahead_tokens(13)

model = AutoModelForRNNT.from_pretrained(
    MODEL_ID,
    trust_remote_code=True,
    device_map="cuda:0",
).eval()

audio, sr = sf.read("audio.wav", dtype="float32")
if audio.ndim > 1:
    audio = audio.mean(axis=1)

inputs = processor(
    audio,
    sampling_rate=sr,
    return_tensors="pt",
    language="vi-VN",  # use "en-US" for English, or "auto" if language is unknown
)
inputs["num_lookahead_tokens"] = 13

dtype = next(model.parameters()).dtype
inputs = {
    key: value.to(model.device, dtype=dtype) if torch.is_floating_point(value) else value.to(model.device)
    for key, value in inputs.items()
}

with torch.inference_mode():
    generated = model.generate(**inputs, max_new_tokens=512)

print(processor.decode(generated.sequences[0], skip_special_tokens=True).strip())

Important settings:

—Use processor.set_num_lookahead_tokens(13) and pass num_lookahead_tokens=13.
—Use max_new_tokens=512; smaller caps may truncate long utterances.
—Prefer explicit language prompts such as vi-VN or en-US when known. auto works but was slightly worse.

Results Before And After Fine-Tuning

Vietnamese scores below are normalized with the repo ASR normalizer, including spoken-number handling. The base model is nvidia/nemotron-3.5-asr-streaming-0.6b. The fine-tuned model is this encoder fine-tune with its RNNT decoder converted to native Transformers. The repository name may contain encoder-only for historical reasons, but the uploaded model is loadable with AutoModelForRNNT.

Dataset	Mode	WER	CER	Exact
PW733 406	Base NeMo streaming `[56,13]`	35.62%	21.18%	23.15%
PW733 406	Fine-tuned NeMo streaming `[56,13]`	13.65%	7.20%	56.90%
PW733 406	Fine-tuned native Transformers, lookahead 13	14.82%	8.34%	n/a
LSVSC 1000	Base NeMo streaming `[56,13]`	11.18%	8.00%	28.10%
LSVSC 1000	Fine-tuned NeMo streaming `[56,13]`	7.69%	4.61%	33.90%
LSVSC 1000	Fine-tuned native Transformers, lookahead 13	7.62%	4.55%	34.10%
HeySQuAD 100 curated local manifest	Native Transformers, `en-US`	11.89%	5.25%	n/a
HeySQuAD 100 curated local manifest	Native Transformers, `auto`	12.18%	5.41%	n/a
HeySQuAD remote validation, first 100	Native Transformers, `en-US`	22.56%	8.46%	n/a
HeySQuAD remote validation, first 200	Native Transformers, `en-US`	27.29%	n/a	n/a

Summary:

—PW733 normalized WER improved from 35.62% to 14.82% on the reproducible 406-file evaluation command below (342 / 2308 word errors; 836 / 10023 character errors).
—LSVSC normalized WER improved from 11.18% to about 7.6%.
—The converted native Transformers checkpoint matches the fine-tuned NeMo checkpoint when evaluated with num_lookahead_tokens=13 and max_new_tokens=512.
—The 11.89% English score uses a curated local 100-sample manifest. It must not be compared directly with the remote validation split, which produced about 20-27% WER in the checks above.

Upload

bash

huggingface-cli upload \
  YOUR_USERNAME/YOUR_REPO_NAME \
  checkpoints/hf_nemotron35_encoder_only_0992_from_nemo \
  . \
  --repo-type model