matbee/LFM2.5-Audio-1.5B-ONNX-cuda-gqa

LFM2.5-Audio-1.5B-ONNX (CUDA-clean GQA patch + sampling-capable unrolled depthformer)

This is a derivative of `LiquidAI/LFM2.5-Audio-1.5B-ONNX` containing only the depthformer graph files that differ from upstream. It does NOT re-host the large .onnx_data sidecar tensors — clone the upstream repo first and overlay these files.

What's different

The upstream depthformer's GroupQueryAttention (com.microsoft) nodes:

• —Wire cos/sin tensors as required inputs
• —Don't set do_rotary / local_window_size / softcap attributes
• —FP16: routes to CUDA EP, dereferences cos/sin regardless of do_rotary → cudaErrorIllegalAddress at /layers.0/gqa
• —Q4: falls back to CPU EP with 372 MemcpyToHost / MemcpyFromHost nodes (slow)

All four files in this repo:

• —Clear the cos / sin GQA inputs (they were dead weight — the upstream graph pre-applies the RoPE layout transform via rearrange_interleaved_to_split)
• —Explicitly set do_rotary=0, local_window_size=-1, softcap=0.0
• —Drop the now-unused gqa_cos / gqa_sin initializers
• —Apply to either the rolled (If-subgraph) or unrolled-8-codebook graph

Plus two new variants do in-graph top-k temperature sampling via the Gumbel-max trick — no CPU↔GPU round-trips per codebook step, fully sampled (NOT greedy argmax).

Files

Model artifacts (onnx/)

All files use the same external-data sidecars (.onnx_data) as upstream — drop them in place on top of an upstream clone.

Reproduction scripts (tools/)

How to rebuild from upstream

# 1. Clone upstream (gets the heavy .onnx_data sidecars)
hf download LiquidAI/LFM2.5-Audio-1.5B-ONNX --local-dir LFM2.5-Audio-1.5B-ONNX

# 2. Clone this repo (gets the tool scripts and the pre-built artifacts)
hf download matbee/LFM2.5-Audio-1.5B-ONNX-cuda-gqa --local-dir cuda-gqa
pip install onnx numpy

# 3. Patch the upstream FP16 graphs (the .onnx_data sidecars are NOT modified)
python cuda-gqa/tools/patch_depthformer_gqa.py \
  --input  LFM2.5-Audio-1.5B-ONNX/onnx/vocoder_depthformer_fp16.onnx \
  --output LFM2.5-Audio-1.5B-ONNX/onnx/vocoder_depthformer_fp16.onnx
python cuda-gqa/tools/patch_depthformer_gqa.py \
  --input  LFM2.5-Audio-1.5B-ONNX/onnx/vocoder_depthformer_fp16_argmax.onnx \
  --output LFM2.5-Audio-1.5B-ONNX/onnx/vocoder_depthformer_fp16_argmax.onnx

# 4. Build the sampling-capable unrolled variants
python cuda-gqa/tools/build_lfm25_depthformer_unrolled_sample.py \
  --input  LFM2.5-Audio-1.5B-ONNX/onnx/vocoder_depthformer_fp16.onnx \
  --output LFM2.5-Audio-1.5B-ONNX/onnx/vocoder_depthformer_fp16_unrolled_sample.onnx \
  --temperature 1.0 --top-k 4
python cuda-gqa/tools/build_lfm25_depthformer_unrolled_sample.py \
  --input  LFM2.5-Audio-1.5B-ONNX/onnx/vocoder_depthformer_q4.onnx \
  --output LFM2.5-Audio-1.5B-ONNX/onnx/vocoder_depthformer_q4_unrolled_sample.onnx \
  --temperature 1.0 --top-k 4

Or skip steps 3+4 and just overlay the pre-built .onnx files from this repo onto your upstream clone.

Sampling math (the new _unrolled_sample variants)

For each of the 8 codebook steps:

logits → Mul(1/T)              # temperature scaling
       → TopK(k=4)              # top-k masking, returns (values, indices)
       → RandomUniformLike      # u ~ U(1e-9, 1) on same shape as values
       → Log → Mul(-1) → Log → Mul(-1)   # Gumbel(0,1): g = -log(-log(u))
       → Add(top_k_vals, g)     # perturbed logits
       → ArgMax                 # index in [0, k) — equivalent to categorical sample
       → GatherElements(top_k_idx, argmax)   # recover real token id

This is the Gumbel-max trick: argmax(logits/T + g) where g ~ Gumbel(0,1) is equivalent to sampling from softmax(logits/T). Combined with top-k masking, it gives top-k categorical sampling with no softmax/multinomial ops, all GPU-resident.

Build script: `tools/build_lfm25_depthformer_unrolled_sample.py` in the remotemedia-sdk project. To rebuild with different defaults:

python tools/build_lfm25_depthformer_unrolled_sample.py \
  --input  vocoder_depthformer_fp16.onnx \
  --output vocoder_depthformer_fp16_unrolled_sample.onnx \
  --temperature 1.0 --top-k 4

(Temperature and top-k are baked into the graph as constants — rebuild to change them.)

Benchmarks

Hardware: NVIDIA RTX 4090, ONNX Runtime via the ort Rust crate (CUDA EP). Workload: woodworks_question.wav (a 4.9s "Can you help me come up with a slogan…" prompt), 5 measured utterances + 2 warmup utterances, firstChunkAudioBatchSize=4, audioBatchSize=12, audioTemperature=1.0, audioTopK=4, maxNewTokens=4096. Quality scored by Whisper-large-v3-turbo coherence (jaccard between expected response set and transcribed response).

Reference: vs Python liquid_audio (LFM2-Audio-1.5B)

Same input prompt, same model weights, same conversational target. Both speed (2.3× faster TTFA) and quality (2× higher jaccard) improved.

Three-way precision comparison

Memcpy boundaries (ORT graph warnings at session-build time)

The 372 Memcpy nodes in the Q4 depthformer come from Q4-quantized MatMul ops which lack full CUDA kernel coverage in ONNX Runtime — the GQA patch in this repo only fixes the attention boundary. Hybrid mode replaces just the depthformer with FP16 to eliminate those 372 Memcpy from the inner autoregressive loop, while keeping Q4's smaller RAM footprint everywhere else.

Sample responses (same input prompt across all three configs)

Full Q4 (jaccard 0.82, smallest footprint):

• —"Sure. Do you prefer a nature-focused vibe or a modern design focus?"
• —"How about crafting beauty, one cut at a time?"
• —"Create better homes, greener work. Does that capture the vibe you want?"

Hybrid Q4 + FP16 depthformer (best TTFA p50, 21.4 words/utt):

• —"Try in Woodcrafted Beauty, where each piece sings from original wood. How does that feel for your brand?"
• —"Join the craft lovers with precision and style at Mosaic Basics Woodworking. Does that capture the vibe anyone?"
• —"Sure thing. How about something like handcrafted wood, precision made great way..."

Full FP16 (lowest wall clock, cleanest CUDA placement):

• —"How about handcrafted woodworking creates your dream? Or handmade woodworking turning home's heart's design?"
• —"Sure thing. How about... Handmade with love and power."
• —"How about adorn your home with timeless craft?"

Recommendation by VRAM budget

Manifest examples for each config are in the remotemedia-sdk repo under examples/manifests/lfm2_onnx_audio_s2s_bench_*.json.

Usage

Drop these files on top of a fresh LiquidAI/LFM2.5-Audio-1.5B-ONNX clone:

git clone https://huggingface.co/LiquidAI/LFM2.5-Audio-1.5B-ONNX
git clone https://huggingface.co/matbee/LFM2.5-Audio-1.5B-ONNX-cuda-gqa
cp -r LFM2.5-Audio-1.5B-ONNX-cuda-gqa/onnx/* LFM2.5-Audio-1.5B-ONNX/onnx/
# now use LFM2.5-Audio-1.5B-ONNX as you would the upstream repo

Or, with hf (Hugging Face Hub CLI):

hf download LiquidAI/LFM2.5-Audio-1.5B-ONNX --local-dir LFM2.5-Audio-1.5B-ONNX
hf download matbee/LFM2.5-Audio-1.5B-ONNX-cuda-gqa --local-dir LFM2.5-Audio-1.5B-ONNX-cuda-gqa
cp -r LFM2.5-Audio-1.5B-ONNX-cuda-gqa/onnx/* LFM2.5-Audio-1.5B-ONNX/onnx/

License

This is a derivative of LiquidAI/LFM2.5-Audio-1.5B-ONNX, distributed under LFM Open License v1.0.

See NOTICE.md for the modification notice required by §4(b).

Acknowledgements

Original model + ONNX export by Liquid AI. Patches & sampling variants by acidhax/remotemedia-sdk.