Firworks/Step-Audio-R1-nvfp4

1---2datasets:3- Rombo-Org/Optimized_Reasoning4base_model:5- stepfun-ai/Step-Audio-R16license: apache-2.07---8# Step-Audio-R1-nvfp49 10**Format:** NVFP4 — weights & activations quantized to FP4 with dual scaling.  11**Base model:** `stepfun-ai/Step-Audio-R1`  12**How it was made:** One-shot calibration with LLM Compressor (NVFP4 recipe), long-seq calibration with Rombo-Org/Optimized_Reasoning.  13 14> Notes: Keep `lm_head` in high precision; calibrate on long, domain-relevant sequences.15 16# 📘 About This Model17 18This is a quantized NVFP4 (W4A4) version of Step-Audio-R1, an open-weights Audio–based multimodal model for audio understanding and reasoning.19The original BF16 model requires ~67 GB VRAM.20 21Step-Audio-R1 combines:22A high-capacity audio encoder23A projection layer that maps audio features into the transformer24A language backbone for reasoning and text generation25 26The model is designed for:27Speech transcription and interpretation28Emotional / tonal analysis29Speaker characteristics30Music and sound-scene understanding31High-quality step-by-step reasoning about audio inputs32 33It does not generate audio; it produces text based on audio input.34 35# 📦 What This Quantized Version Enables36 37This NVFP4 quantized version reduces memory requirements significantly:38Size: ~22 GB (down from ~67 GB)39Should fit comfortably on a single RTX 509040 41Preserves most reasoning performance from the BF16 release42Because of this, anyone with a high-end consumer GPU can experiment with advanced audio reasoning locally.43 44 45Check the original model card for more information about this model.46 47# Running the model with VLLM in Docker48It requires a specific vllm container released by the model authors.49```sh50docker run --rm -ti --gpus all \51    -v $(pwd)/Step-Audio-R1:/Step-Audio-R1 \52    -p 9999:9999 \53    stepfun2025/vllm:step-audio-2-v20250909 \54    vllm serve /Step-Audio-R1 \55    --served-model-name Step-Audio-R1 \56    --port 9999 \57    --max-model-len 16384 \58    --max-num-seqs 32 \59    --chat-template '{%- macro render_content(content) -%}{%- if content is string -%}{{- content.replace("<audio_patch>\n", "<audio_patch>") -}}{%- elif content is mapping -%}{{- content['"'"'value'"'"'] if '"'"'value'"'"' in content else content['"'"'text'"'"'] -}}{%- elif content is iterable -%}{%- for item in content -%}{%- if item.type == '"'"'text'"'"' -%}{{- item['"'"'value'"'"'] if '"'"'value'"'"' in item else item['"'"'text'"'"'] -}}{%- elif item.type == '"'"'audio'"'"' -%}<audio_patch>{%- endif -%}{%- endfor -%}{%- endif -%}{%- endmacro -%}{%- if tools -%}{{- '"'"'<|BOT|>system\n'"'"' -}}{%- if messages[0]['"'"'role'"'"'] == '"'"'system'"'"' -%}{{- render_content(messages[0]['"'"'content'"'"']) + '"'"'<|EOT|>'"'"' -}}{%- endif -%}{{- '"'"'<|BOT|>tool_json_schemas\n'"'"' + tools|tojson + '"'"'<|EOT|>'"'"' -}}{%- else -%}{%- if messages[0]['"'"'role'"'"'] == '"'"'system'"'"' -%}{{- '"'"'<|BOT|>system\n'"'"' + render_content(messages[0]['"'"'content'"'"']) + '"'"'<|EOT|>'"'"' -}}{%- endif -%}{%- endif -%}{%- for message in messages -%}{%- if message["role"] == "user" -%}{{- '"'"'<|BOT|>human\n'"'"' + render_content(message["content"]) + '"'"'<|EOT|>'"'"' -}}{%- elif message["role"] == "assistant" -%}{{- '"'"'<|BOT|>assistant\n'"'"' + (render_content(message["content"]) if message["content"] else '"'"''"'"') -}}{%- set is_last_assistant = true -%}{%- for m in messages[loop.index:] -%}{%- if m["role"] == "assistant" -%}{%- set is_last_assistant = false -%}{%- endif -%}{%- endfor -%}{%- if not is_last_assistant -%}{{- '"'"'<|EOT|>'"'"' -}}{%- endif -%}{%- elif message["role"] == "function_output" -%}{%- else -%}{%- if not (loop.first and message["role"] == "system") -%}{{- '"'"'<|BOT|>'"'"' + message["role"] + '"'"'\n'"'"' + render_content(message["content"]) + '"'"'<|EOT|>'"'"' -}}{%- endif -%}{%- endif -%}{%- endfor -%}{%- if add_generation_prompt -%}{{- '"'"'<|BOT|>assistant\n<think>\n'"'"' -}}{%- endif -%}' \60    --enable-log-requests \61    --interleave-mm-strings \62    --trust-remote-code63```64 65This example script should allow an audio wave file to be streamed to the model and get a response based on the prompt.66```py67import requests68import base6469 70with open("audio.wav", "rb") as f:71    audio_b64 = base64.b64encode(f.read()).decode()72 73payload = {74    "model": "Step-Audio-R1",75    "stream": True,76    "messages": [77        {78            "role": "user",79            "content": [80                {"type": "input_audio", "audio_data": audio_b64, "mime_type": "audio/wav"},81                {"type": "text", "text": "Transcribe this and describe the speaker."}82            ]83        }84    ]85}86 87with requests.post(88    "http://localhost:9999/v1/chat/completions",89    json=payload,90    stream=True,91) as r:92    for line in r.iter_lines():93        if line:94            print(line.decode())95```96 97This was tested on an RTX Pro 6000 Blackwell cloud instance.98 99If there are other models you're interested in seeing quantized to NVFP4 for use on the DGX Spark, or other modern Blackwell (or newer) cards let me know. I'm trying to make more NVFP4 models available to allow more people to try them out.