mbehr90/whisper-large-v3-turbo-nvfp4

whisper-large-v3-turbo — NVFP4

NVFP4 on all 232 language-model Linear layers, produced with llm-compressor 0.13.0 / compressed-tensors 0.18.0 and measured on an NVIDIA H100 80 GB.

0.53 GiB on disk, from 1.51 GiB.

What is quantized

FP4 E2M1 weights with an FP8 E4M3 scale per 16 values and one FP32 global scale per module; activations quantized per 16-value group at runtime. proj_out and the encoder Conv1d layers are untouched.

Smallest of the three at 0.53 GiB, at a cost of about 11 % relative WER (2.561 % against 2.303 % under transformers). On Hopper vLLM runs NVFP4 through Marlin as a W4A16 GEMM, so there is no FP4 arithmetic to gain from; on Blackwell the same checkpoint runs true W4A4. Read the throughput column before deploying this on Hopper.

Measured on H100

WER on the first 500 utterances of LibriSpeech test-clean, scored after Whisper's EnglishTextNormalizer, identical corpus and identical WER implementation for both runtimes. transformers loads the checkpoint with run_compressed=False (decompressed to bf16), so that column isolates the quantization from the kernel. RTFx = seconds of audio transcribed per second of wall clock, 16 concurrent requests.

Honest summary: on H100 this is not a win

Quantization makes this model slower, not faster. RTFx measured on the same 500 utterances at 16 concurrent requests: fp16 437.8, FP8 317.1 (−28 %), NVFP4 380.5 (−13 %).

The reason is size. At 1.6 GB there is no memory-bandwidth pressure to relieve, the workload is encoder-bound and compute-bound, and with d_model = 1280 the GEMMs are small enough that per-matmul dequantization overhead outweighs what the FP8 tensor cores give back. The whole model comfortably fits in HBM either way; the absolute saving is under one gigabyte.

Use this build if you are packing many Whisper instances onto one card, or targeting Blackwell where NVFP4 runs as true W4A4. For a single-model H100 deployment, the fp16 original is faster and at least as accurate.

Why the encoder/decoder split matters

whisper-large-v3-turbo is lopsided: 32 encoder layers (192 `Linear`, ~630M params) against 4 decoder layers (40 `Linear`, ~105M). A third build, NVFP4 on the encoder and FP8 on the decoder, scores 2.422 % — meaning the 4-bit decoder, at 13 % of the weights, costs roughly as much accuracy as the entire 4-bit encoder does. Four layers that run autoregressively per token are simply more sensitive than 32 that run once per 30 s window.

That mixed build is not published: no vLLM available here can load it. Whisper's packed_modules_mapping fuses cross-attention K and V into a two-way MergedColumnParallelLinear, and the mixed-precision weight loader dies on it with AttributeError: 'MergedColumnParallelLinear' object has no attribute 'data' — in 0.26.0 and in a 0.1.dev20073 nightly alike. The same mixed-precision format loads fine for decoder-only models.

Serving

vllm serve mbehr90/whisper-large-v3-turbo-nvfp4 --max-model-len 448
curl http://localhost:8000/v1/audio/transcriptions \
  -F file=@audio.wav -F model=mbehr90/whisper-large-v3-turbo-nvfp4 -F language=en

The stock vllm/vllm-openai image ships without audio extras — every transcription request comes back 400 with "Please install vllm[audio] for audio support". Install soundfile and librosa into the image, or into a directory on PYTHONPATH.

Reproducing

recipe.yaml is the exact llm-compressor recipe. Calibration for NVFP4: 256 LibriSpeech utterances as log-mel input_features plus decoder_input_ids. Two things that do not work out of the box — llm-compressor's dataset path assumes text (data["labels"] = data["input_ids"].copy(), and Whisper has no input_ids), so pass a ready DataLoader; and the sequential pipeline cannot torch.fx-trace an encoder-decoder, so use pipeline="basic".