CoolFace
Modelpublic

Ar4ikov/VibeVoice-ASR-Streaming-1.5B-DFlash2-Drafter-AWQ-W4A16-ASYM

sourceHugging Facemitupdated 2d agoView on Hugging Face
0likes25downloads
Model Card

VibeVoice-ASR-Streaming-1.5B-DFlash2-Drafter-AWQ-W4A16-ASYM

A DFlash 2 block drafter for VibeVoice-ASR-Streaming-1.5B (Ar4ikov/VibeVoice-ASR-Streaming-1.5B-AWQ-W4A16-ASYM), for speculative decoding in vibevoice.c. It proposes 8 tokens in one pass; vibevoice.c checks them in one pass of the model and keeps the ones the model agrees with, plus one of its own.

The check is exact: every checked row is computed with the arithmetic of the model's own one-token decode step, so the transcript with this drafter is byte-for-byte the transcript without it — words, timestamps, speakers. The drafter only changes how many passes it takes.

This is the drafter with its projections stored as INT4 (AWQ, compressed-tensors pack-quantized, W4A16 asymmetric, groups of 128): 0.23 GB instead of the BF16 checkpoint's (Ar4ikov/VibeVoice-ASR-Streaming-1.5B-DFlash2-Drafter). vibevoice.c loads the codes as they are. Accepted drafts on the held-out traces: 3.620 tokens per block, against 3.616 in BF16.

Use

Needs vibevoice.c with DFlash 2 support: branch dflash2 (PR #48), in the next release. The model and this drafter in one download: Ar4ikov/VibeVoice-ASR-Streaming-1.5B-AWQ-W4A16-ASYM-DFlash2 (the INT4 drafter in drafter/, used without --draft).

bash
vv_cli --model ./VibeVoice-ASR-Streaming-1.5B-AWQ-W4A16-ASYM --audio talk.wav --draft ./VibeVoice-ASR-Streaming-1.5B-DFlash2-Drafter-AWQ-W4A16-ASYM
vv_cli serve --model ./VibeVoice-ASR-Streaming-1.5B-AWQ-W4A16-ASYM --draft ./VibeVoice-ASR-Streaming-1.5B-DFlash2-Drafter-AWQ-W4A16-ASYM --slots 4           # streaming sessions (WebSocket, SSE) too
flag
--draft <dir>this repository
--draft-block <n>rows of the drafted block the model checks per pass (the drafter drafts 8; without the flag: 4 or 8, whichever keeps more tokens per ms)
`--draft-check exact\fast`exact (default): the transcript without a drafter on the same --attn, bit for bit; fast: flashinfer attention and prefill projections for the check, the greedy transcript within rounding
`--draft-quant int4\f16`how the runtime holds the drafter (this one is INT4 already)

It drafts for any VibeVoice-ASR-Streaming-1.5B checkpoint of the same model — microsoft/VibeVoice-ASR-Streaming-1.5B — since it reads only the model's hidden states, embedding and LM head; it was trained on the AWQ checkpoint's transcripts, so that is where it drafts best.

Results

vibevoice.c 7d7d43e (branch dflash2), RTX 3090, VibeVoice-ASR-Streaming-1.5B-AWQ-W4A16-ASYM, greedy decoding, decode tokens per second:

plaindraftedspeeduptokens per blocksame transcript
20 held-out clips, 8 rows409 tok/s866 tok/s2.12x3.5320/20
2-minute file, 8 rows416 tok/s1047 tok/s2.52x4.03yes
32-minute file, 8 rows341 tok/s722 tok/s2.12x4.03yes

The same clips as the BF16 drafter's card.

The drafter

  • —DFlash 2: 5 Qwen3-style layers (hidden 1536, 12/2 heads, intermediate 4480), 227M parameters, block 8.
  • —KV injection: for every position the model has processed, the outputs of its layers 1, 7, 13, 19 and 25 are concatenated, projected (fc) and normalized; every drafter layer turns them into keys and values. The drafter reads the model's own view of the audio and of the transcript so far.
  • —Two-tap dynamic convolution around attention and MLP (per-row kernels predicted from the row), and a candidate selector that re-ranks the model head's top 16 per row with a pairwise predecessor/successor term (rank 256).
  • —The embedding and the LM head are the model's, frozen. The drafter scores a draft vocabulary of 32768 ids (draft_vocab): the ids its training transcripts use, plus every stop id.
  • —Accepted drafts on 43 held-out clips' traces: 3.620 tokens per block of 8 (the selector's, with every earlier draft right).

Training

Self-distillation: the drafter learns what this model writes, timestamps and speaker ids included. tools/dflash in vibevoice.c holds the pipeline:

  1. 1.~140 hours of audio, 2872 training clips and 43 held out, from LibriSpeech, FLEURS (8 languages), SOVA, AMI, earnings calls and VoxConverse — audio only; the datasets' own text is never used.
  2. 2.The model's own greedy transcripts of every clip (vv_dflash_data gen).
  3. 3.Traces: every position's token, its role and the five tapped layers' outputs, replayed through the runtime as streaming sessions (vv_dflash_data trace).
  4. 4.train.py (PyTorch, flex attention): anchors at generated positions, rows 1..7 scored against the next 7 tokens with weights e^(-k/4); cross-entropy of the head plus the selector's; AdamW, cosine schedule, BF16.
  5. 5.awq_drafter.py: activation-aware scales where they fold into the drafter without changing it (the MLP input, up → down, v → o per KV channel), then a clip search on every projection; INT4 codes stored as compressed-tensors.

Limits

  • —vibevoice.c only: the checkpoint keeps the published DFlash 2 names, but the KV injection, the selector and the draft vocabulary are as vibevoice.c implements them (docs/DFLASH.md).
  • —CUDA only (the CPU path and Metal decode without the drafter). Greedy only.
  • —Audio the corpus covered little (Mandarin, Russian) keeps fewer drafts; the runtime then falls back to plain steps and stays near plain speed.

License

MIT, like VibeVoice.