nvidia/NeMo-Speech-Streaming-Multispeaker-ASR

8-Speaker Streaming Transcription Demo

This Gradio application combines Nemotron 3.5 streaming ASR with Nemotron diarization. It produces speaker-attributed transcripts and speaker activity maps for uploaded files or live microphone audio, including overlapping speech with up to eight speakers.

🚀 Features

• —File playback synchronized with streaming ASR and diarization
• —Live microphone transcription
• —Up to eight color-coded speakers
• —Overlap-aware speaker activity visualization
• —Automatic transcript scrolling
• —Configurable chunk size, speaker count, silence breaks, cache gating, and diarization binarization
• —BF16 inference on supported CUDA GPUs

📋 Requirements

• —A CUDA-capable GPU
• —The latest `NVIDIA-NeMo/Speech` main branch
• —Gradio 6.x
• —Two compatible checkpoints:
• —Nemotron 3.5 streaming ASR
• —Nemotron diarization

🛠️ Installation

Clone NeMo Speech:

git clone --branch main --single-branch https://github.com/NVIDIA-NeMo/Speech.git NeMo-Speech
cd NeMo-Speech

Then create a Python 3.13 Conda environment:

conda create --name nemo_py313 python=3.13 -y
conda activate nemo_py313
python -m pip install --upgrade pip
python -m pip install uv

# Install the project into the active Conda environment.
export UV_PROJECT_ENVIRONMENT="$CONDA_PREFIX"
uv sync --extra all --extra cu13     # CUDA 13.x (recommended) — use --extra cu12 for CUDA 12.x
uv pip install "gradio>=6,<7"

The default checkpoints in run_app.sh use local model paths:

• —/pool/hf_models/nemotron-3.5-asr-streaming-0.6b/nemotron-3.5-asr-streaming-0.6b.nemo
• —/pool/hf_models/Nemotron-3-Diarization/Nemotron-3-Diarization.nemo

🎯 Usage

Start the app from the demo directory. With no model arguments, it uses the defaults configured in app.py:

python app.py

To use different models, pass a Hugging Face repository ID or a local checkpoint path for either option:

python app.py \
  --diar_model /path/to/Nemotron-3-Diarization.nemo \
  --asr_model /path/to/nemotron-3.5-asr-streaming-0.6b.nemo \
  --target_lang auto

The app listens on port 7861 and requests a Gradio share URL.

Alternatively, update the environment and NeMo root in run_app.sh, then run:

chmod +x run_app.sh
./run_app.sh

🖥️ Use the interface

🎙️ Microphone

The Microphone tab opens by default.

1. 1.Start recording.
2. 2.Speak normally; speaker activity and transcript updates appear as audio is processed.
3. 3.Use Pause / Resume beside Stop to pause or continue transcription.
4. 4.Stop recording to reset the microphone session.

📁 File upload

1. 1.Select the File Upload tab.
2. 2.Upload an audio file, or select Choose Example to load the bundled sample.
3. 3.Press Play to start synchronized playback and processing.
4. 4.Use the same control to pause or resume playback.
5. 5.Select ✕ beside the playback control before loading another file.

Clearing a file also clears its transcript and speaker activity map. A newly loaded file starts at 0:00. Switching between Microphone and File Upload stops active processing and resets both streaming instances. Additional supported audio files placed in audio_data/ appear in the example picker automatically.

⚙️ Default configuration

• —Maximum speakers: 8
• —Streaming chunk size: 13 ASR frames (1.12 seconds per advance)
• —Text break silence: 1.0 second
• —Cache gating: enabled
• —Binary diarization decisions: enabled
• —ASR attention context: [56, 13]
• —Sortformer speaker cache: inherited from the diarization checkpoint
• —Sortformer cache update period: 222 frames
• —Sortformer FIFO: 264 frames
• —Diarization output subsampling factor: matched to the ASR encoder (8 for the default model)
• —Sample rate: 16 kHz
• —Precision: BF16 when supported

Changing a control reloads both streaming sessions with the selected settings.

🤖 Model compatibility

Compatible diarization model:

• —nvidia/Nemotron-3-Diarization (default, up to 8 speakers)

Compatible ASR model:

• —nvidia/nemotron-3.5-asr-streaming-0.6b (default)

📂 Files

app.py               Gradio UI, callbacks, and browser-side behavior
init_nemo_models.py  Model loading and streaming session integration
gradient_html.py     Eight-speaker activity visualization and colors
space_article.html   Compact application header
run_app.sh           Local launch example
audio_data/          Bundled example audio
README.md            This document

🔧 Troubleshooting

• —Model not found: Verify local checkpoint paths or Hugging Face repository IDs.
• —NeMo import error: Ensure the repository root is present in PYTHONPATH.
• —CUDA out of memory: Close other GPU workloads and restart the app.
• —No share URL: The app retries transient tunnel failures; use the local URL if the share service remains unavailable.
• —No microphone found: Use the HTTPS Gradio share URL or localhost, allow microphone access in the browser, and verify that the operating system exposes an input device. Browsers block microphone access on insecure remote HTTP URLs such as http://server-ip:7861.
• —Unsupported audio: Convert the input to a common format such as 16 kHz WAV.