suryatmodulus/Audio8-ASR-Infinite

<div align="center">

Audio8 ASR Infinite

   

</div>

Audio8 ASR Infinite is a native streaming speech recognition model built to be as responsive as possible. It offers a selectable audio clock (80/120/160 ms) and a transcription delay (240–560 ms). With our adapted vLLM build it transcribes unlimited-length audio 24/7 without drifting.

Highlights

• —Super responsive — the native streaming architecture decodes 12.5 times per second.
• —Unlimited-length transcription — a rolling KV cache keeps memory and latency bounded, even in 24/7 operation.
• —Selectable streaming clock — one text token per clock step (12.5 / 8.3 / 6.25 decisions per second), balancing perception granularity and resource cost.
• —Configurable transcription delay — set how much delay to trade for accuracy.
• —Semantic VAD — distinguishes thinking pauses, stuttering and real end of turn, where traditional acoustic VAD fails.
• —Bilingual — Chinese and English.

Optimized operation points

The following combinations of frame length and delay are post-trained. Other combinations can be used but performance may not be optimum.

target_delay_ms must be an integer multiple of the selected clock, so longer delays stay available at every clock even when they are not listed above.

Architecture

Inherits the Voxtral realtime audio architecture and DSM-style streaming.

Checkpoint specification:

Roadmap

This is the preview release: it delivers the transcription base. Realtime semantic perception is being built on the same frame grid and the same acoustic forward pass.

Evaluation

480 ms Delay, 80ms frame length

Greedy decode with EOS suppressed, at the 80 ms audio clock with target_delay_ms = 480 (6 delay tokens). Error rates in percent. No repetition loops and no dropped trailing words.

Usage

Programmatic simulated-streaming decode with the embedded remote code:

import numpy as np
import torch
from transformers import AutoFeatureExtractor, AutoTokenizer

from audio8_asr_infinite.modeling.modeling_audio8_asr_infinite import (
    Audio8ASRInfiniteForConditionalGeneration,
    resolve_qwen_language_token_id,
    resolve_qwen_streaming_special_token_ids,
)
from audio8_asr_infinite.streaming_inference import simulated_streaming_greedy_decode_batch

checkpoint = "Edge0/Audio8-ASR-Infinite"
tokenizer = AutoTokenizer.from_pretrained(checkpoint, trust_remote_code=True)
feature_extractor = AutoFeatureExtractor.from_pretrained(checkpoint, trust_remote_code=True)
model = Audio8ASRInfiniteForConditionalGeneration.from_pretrained(
    checkpoint, trust_remote_code=True, torch_dtype=torch.bfloat16
).eval().cuda()

class AudioConfig:  # duck-typed: raw_audio_samples_per_token / streaming_n_left_pad_tokens / sampling_rate
    raw_audio_samples_per_token = 1280   # 80 ms @ 16 kHz
    streaming_n_left_pad_tokens = 18
    sampling_rate = 16000

waveform = np.load("sample.npy", allow_pickle=False).astype(np.float32)  # [-1, 1], 16 kHz mono
results = simulated_streaming_greedy_decode_batch(
    model=model,
    tokenizer=tokenizer,
    feature_extractor=feature_extractor,
    waveforms=[waveform],
    language_token_ids=[resolve_qwen_language_token_id(tokenizer, "zh")],
    special_ids=resolve_qwen_streaming_special_token_ids(tokenizer),
    audio_config=AudioConfig(),
    num_delay_tokens=[480 // 80],
    right_pad_text_tokens=10,
    dtype=torch.bfloat16,
    device=next(model.parameters()).device,
    max_new_tokens=512,
)
print(results[0]["final_text"])

Only a full merged weight directory is supported (this repository as-is); adapter-style or partially converted weights are not.

24/7 inference with vLLM

Docker compose is the canonical deployment path; it also serves the web demo:

cd docker
AUDIO8_MODEL_DIR=/path/to/checkpoint docker compose up -d

Verify with the web client shipped in the same stack:

http://localhost:8080/     # plain HTTP
https://localhost:8443/    # TLS proxy; accept the self-signed certificate

The same socket can be driven from a terminal:

python -m audio8_asr_infinite.examples.vllm_realtime_client \
    --ws-url ws://127.0.0.1:18191/v1/realtime \
    --audio sample.wav --language zh --target-delay-ms 480 --pace

18191 is the host port published by docker/docker-compose.yml; the service itself listens on 18190 inside the compose network. The rolling KV window is 30 s with exact RoPE re-basing, which is what keeps memory and latency bounded over 24/7 operation.

Torch inference (simulated streaming decode)

python -m audio8_asr_infinite.examples.torch_streaming_decode \
    --checkpoint /path/to/checkpoint \
    --audio sample.wav --language zh --transcription-delay-ms 480