CoolFace
Modelpublic

giangndm/nemotron-3.5-asr-streaming-decoder

sourceHugging Facecc-by-4.0updated 13d agoView on Hugging Face
0likes43downloads
Model Card

Nemotron-3.5 ASR Streaming Decoder & Projector

This repository contains the standalone Language Projector + RNN-T Predictor + Joint Network extracted from NVIDIA's nvidia/nemotron-3.5-asr-streaming-0.6b.

It accepts raw continuous 1024-dimensional acoustic representations from the Pure Audio Encoder (`giangndm/nemotron-3.5-asr-streaming-encoder`), conditions them on the language prompt, and autoregressively decodes them into text tokens.

Architecture Specifications

ParameterValueDescription
Model TypeLanguage Projector + RNN-T2-layer MLP Projector + 2-layer LSTM Decoder + Joint Head
Total Parameters28.9MProjectors: 5.1M, LSTM Decoder: 15.4M, Joint: 8.4M
Input Acoustic Dim1024Accepts raw continuous speech features directly from Pure Encoder
Projected Dim640Fuses language ID one-hot (prompt_ids) and projects 1024 -> 640
Predictor LSTM2 layersHidden dimension: 640, Embedding: 13,088 $ imes$ 640
Vocabulary Size13,088Multilingual BPE tokens + language prompts
Blank Token ID13087Emission indicating progression along acoustic time axis

Quickstart: End-to-End Speech Transcription

python
import soundfile as sf
import torch
from transformers import AutoFeatureExtractor, AutoModel, AutoTokenizer

enc_repo = "giangndm/nemotron-3.5-asr-streaming-encoder"
dec_repo = "giangndm/nemotron-3.5-asr-streaming-decoder"

# 1. Load Pure Encoder and Decoder
feat_extractor = AutoFeatureExtractor.from_pretrained(enc_repo)
encoder = AutoModel.from_pretrained(enc_repo, trust_remote_code=True, torch_dtype=torch.bfloat16).cuda().eval()

decoder = AutoModel.from_pretrained(dec_repo, trust_remote_code=True, torch_dtype=torch.bfloat16).cuda().eval()
tokenizer = AutoTokenizer.from_pretrained(dec_repo, trust_remote_code=True)

# 2. Process audio
audio, sr = sf.read("voice_sample.wav")
inputs = feat_extractor(audio, sampling_rate=16000, return_tensors="pt")
input_features = inputs.input_features.to("cuda", dtype=torch.bfloat16)

with torch.no_grad():
    # Step 1: Pure acoustic encoding -> (B, T, 1024)
    enc_out = encoder(input_features, num_lookahead_tokens=13)
    acoustic_1024d = enc_out.last_hidden_state

    # Step 2: Decoder projects 1024d -> 640d with language prompt & greedy decodes
    # prompt_ids: 33 for vi-VN, 0 for en-US, 101 for auto
    token_ids = decoder.greedy_decode(acoustic_1024d)[0]

text = tokenizer.decode(token_ids, skip_special_tokens=True)
print("Transcript:", text)

Citation & Acknowledgements