Ar4ikov/VibeVoice-ASR-Streaming-7B-AWQ-W4A16-ASYM

VibeVoice-ASR-Streaming-7B AWQ W4A16 ASYM

Activation-aware INT4 quantization of microsoft/VibeVoice-ASR-Streaming-7B, revision 60d858b518b4e19d404af3737f848fc185b30177: the streaming VibeVoice ASR model, which writes text for every 2.93 s of audio as it arrives, with 0.53 s of lookahead.

Weights: 17.348 GB → 7.000 GB (2.48× smaller). Sizes are decimal GB of safetensors, including every retained speech component and the language model.

Part of the VibeVoice-ASR Quantized collection, built the same way as Ar4ikov/VibeVoice-ASR-AWQ-W4A16-ASYM.

Format and scope

• —Genuine activation-aware AWQ scale search using llm-compressor; not RTN or a conversion of an already quantized checkpoint.
• —All 196 Qwen2 attention/MLP projections: 4-bit asymmetric, group size 128, 16-bit activations.
• —40 grid points with duo_scaling="both"; seed 42.
• —Embeddings, speech encoders/connectors, norms, and biases retained in BF16. The LM head is kept in BF16. AWQ group scales stored in FP16 with an exact representability check.
• —Original model.acoustic_tokenizer.decoder.* removed: this audio synthesis decoder is unused by the ASR path. Saves 687.4 MB. This is an ASR-only checkpoint.
• —The calibrated compressed-tensors checkpoint is losslessly repacked into standard AWQ GEMM (qweight, qzeros, scales). Every integer code round-trips exactly and every scale value is preserved. There is no second quantization pass.
• —Tokenizer files and preprocessor_config.json are the base checkpoint's, unchanged (<|text_chunk_end|>, chunk_frames 22, lookahead_frames 4, normalize_audio false).

Calibration

256 sequences, maximum 2048 rows each, calibrated through inputs_embeds so real speech embeddings are in the data:

• —128 complete streaming sessions in the model's own protocol: the streaming prompt, then for every 26-frame window <|object_ref_start|> + speech features + <|object_ref_end|>, the BF16 model's own greedy chunk text, and <|text_chunk_end|> (about 945 rows per session). Audio: LibriSpeech dev-clean, 32 speakers, streams of 15–150 s built from consecutive utterances of one chapter; about a third alternate between two speakers, so speaker turns appear in the data.
• —128 Ultrachat conversations (HuggingFaceH4/ultrachat_200k, train_sft, shuffled with seed 42) for the generic language-model path.

Speech features come from the BF16 checkpoint's encoders run in FP32 with the deterministic acoustic latent (the mean), each window encoded on its own exactly as upstream streaming_generate does.

Validation

12 held-out streams, 9.7 minutes, 1579 reference words, from the 8 LibriSpeech dev-clean speakers excluded from calibration (4 of the streams alternate between two speakers):

Speaker N: labels and punctuation are removed before scoring. This is an English read-speech regression check, not a representative quality benchmark: multilingual accuracy, noisy audio and diarization quality after quantization have not been measured. evaluation.json has every reference and output.

Speed with vibevoice.c (RTX 3090)

Native W4A16 kernels (no dequantization to FP16), whole file streamed chunk by chunk:

Language-model layers on the GPU: 3306 MB (AWQ) vs 12446 MB (BF16).

Usage

from huggingface_hub import snapshot_download
snapshot_download("Ar4ikov/VibeVoice-ASR-Streaming-7B-AWQ-W4A16-ASYM", local_dir="VibeVoice-ASR-Streaming-7B-AWQ-W4A16-ASYM")

With vibevoice.c:

vv_cli --model ./VibeVoice-ASR-Streaming-7B-AWQ-W4A16-ASYM --audio talk.wav --output transcript.json   # prints chunks as they are ready
vv_cli mic --model ./VibeVoice-ASR-Streaming-7B-AWQ-W4A16-ASYM                                         # live microphone
vv_cli serve --model ./VibeVoice-ASR-Streaming-7B-AWQ-W4A16-ASYM --slots 8                             # SSE stream=true and WebSocket /v1/audio/stream

This custom architecture is not supported by plain AutoModelForCausalLM without VibeVoice integration, and vLLM deployment was not tested. The acoustic decoder was pruned, so integrations must tolerate its absent unused weights.

Streaming protocol

Audio is 24 kHz mono and is not loudness-normalized. The prompt is plain text, tokenized without special tokens:

You are a helpful assistant that transcribes audio input into text output. Please transcribe the following audios streamingly with these keys: speaker, content\n

Then, for every window of 22 + 4 frames (83200 samples, advancing by 70400; the last one zero-padded), encoded on its own: prefill <|object_ref_start|>, the 26 speech feature rows, <|object_ref_end|>; decode greedily until <|text_chunk_end|> or <|endoftext|>; feed <|text_chunk_end|>. EOS does not end the session. Speaker turns appear inline as \n Speaker N:.

Reproduction

recipe.yaml is the actual compression recipe. reproduce/ has the whole pipeline (pipeline.sh): data preparation, checkpoint split and calibration sessions (prepare_model.py, stream_ref.py), quantization, exact AWQ repacking, both evaluations and this card. Set VV_QUANT_WORKDIR, VV_QUANT_DATA, VIBEVOICE_SOURCE (Microsoft VibeVoice source checkout) and VV_CLI. Preparation and the PyTorch comparison used torch 2.6.0+cu124 and transformers 4.51.3; the quantization environment is in quantization_environment.json. One NVIDIA RTX 3090 24 GB.

License: MIT, inherited from the base model. No audio or HF credentials are included.