Modelpublic

acul3/stable-audio-3-executorch-qnn-sm8750

sourceHugging Faceotherupdated 3mo agoView on Hugging Face

0likes75downloads

Model Card

Stable Audio 3 (Small-SFX + Small-Music) — ExecuTorch / Qualcomm QNN — SM8750

On-device text→audio for Stable Audio 3 Small-SFX, lowered to the ExecuTorch QNN backend and validated end-to-end on a Snapdragon 8 Elite (SM8750, HTP V79) phone. The diffusion denoiser runs on the NPU; the text encoder and audio decoder run on CPU (XNNPACK).

Prompt → 16 s stereo 44.1 kHz WAV. The included constants reproduce the demo prompt "a dog barking".

These are runtime artifacts (compiled .pte graphs + a prebuilt aarch64 runner), not trainable checkpoints. Source weights derive from Stability-AI/stable-audio-3; see License below.

Pipeline

prompt ─► T5Gemma encoder ─► cross_attn_cond (1,257,768) + global_cond (1,768)
                                      │
         init noise (1,256,174) ─► 8× ping-pong diffusion, each step = DiT(x, σ)
                                      │
                    final latent ─► SAME-S decoder ─► stereo 44.1 kHz audio

L = latent length = audio_samples / 4096. SFX 10 s (+headroom) → L = 174 → 16 s. The DiT is CFG-free at batch 1 (distilled rf_denoiser).

Stage	Backend	File	Validation (cos vs FP32)
T5Gemma encoder + conditioning	XNNPACK CPU, FP16	`sa3_cond.pte` (563 MB)	0.99997
DiT denoiser (run 8×)	QNN / HTP, FP16	`sa3_dit.pte` (940 MB)	0.9984, ~50 ms/step
SAME-S decoder (Conv1d→Conv2d)	XNNPACK CPU, FP32	`sa3_same_decoder.pte` (218 MB)	1.0000

Performance (SM8750, warm)

The SAME-S decoder's lone Conv1d is exported as a 2D convolution (height 1) so XNNPACK delegates and multithreads it instead of falling back to a single-threaded scalar kernel. This is bit-exact (eager max|Δ| = 0.0; a seed-matched on-device A/B gives cosine 1.000000000, max |Δ| = 1 LSB = −90 dBFS) and ~3.4× faster decode.

End-to-end for the 16 s SFX clip: ~6 s total (model load + conditioning ~2.3 s, 8 DiT steps on the NPU ~0.4 s, decode ~2 s). For reference, an FP16 decoder is a size win only — it runs ~3× slower (no fast FP16 conv microkernels on the Oryon CPU).

sa3_cond.pte            # text encoder + conditioning  (XNNPACK, FP16)
sa3_dit.pte             # diffusion denoiser           (QNN/HTP, FP16) — run 8×
sa3_same_decoder.pte    # latent → audio decoder       (XNNPACK, FP32)
sa3_runner              # prebuilt aarch64 ExecuTorch runner (gflags CLI)
consts/
  input_ids.bin         # tokenized prompt "a dog barking" (256 × int64)
  attn.bin              # prompt attention mask            (256 × bool)
  lac.bin padext.bin mem.bin freqs.bin   # L=174 constants (local_add_cond,
                        #   externalized pad mask, memory tokens, RoPE freqs)
lib/
  libqnn_executorch_backend.so   # ExecuTorch QNN backend (BSD)
  libc++_shared.so               # NDK C++ runtime

`input_ids.bin` / `attn.bin` encode the prompt. To synthesize a different prompt, re-tokenize with the T5Gemma (Gemma SentencePiece) tokenizer on a host and overwrite these two blobs (256-length, left/right per the model). lac/padext/mem/freqs.bin are L-dependent constants and are reused for any L=174 prompt.

Requirements

—A Snapdragon 8 Elite / SM8750 device (HTP V79). Other SoCs need the DiT re-lowered with --soc_model for that HTP version.
—The 5 Qualcomm QNN runtime libraries (proprietary — not redistributed here), from the Qualcomm QNN SDK 2.37+:

  libQnnHtp.so
  libQnnHtpPrepare.so
  libQnnHtpV79Skel.so
  libQnnHtpV79Stub.so
  libQnnSystem.so

Copy them from $QNN_SDK_ROOT/lib/aarch64-android/ (and the V79 skel from $QNN_SDK_ROOT/lib/hexagon-v79/unsigned/) into lib/ next to the shipped .so.

Run on device

bash

# push (one transfer; never run concurrent adb pushes)
adb shell mkdir -p /data/local/tmp/sa3
adb push sa3_cond.pte sa3_dit.pte sa3_same_decoder.pte sa3_runner /data/local/tmp/sa3/
adb push consts lib /data/local/tmp/sa3/
# add the 5 Qualcomm .so to lib/ (see Requirements), then:
adb push libQnnHtp.so libQnnHtpPrepare.so libQnnHtpV79Skel.so \
         libQnnHtpV79Stub.so libQnnSystem.so /data/local/tmp/sa3/lib/

adb shell '
  cd /data/local/tmp/sa3
  chmod +x sa3_runner
  LD_LIBRARY_PATH=$PWD/lib ADSP_LIBRARY_PATH=$PWD/lib \
    ./sa3_runner --cond sa3_cond.pte --dit sa3_dit.pte \
                 --decoder sa3_same_decoder.pte --consts consts \
                 --output sa3_out.wav --seed 0'

adb pull /data/local/tmp/sa3/sa3_out.wav .

The runner: loads the conditioning, runs the 8-step ping-pong sampler (fixed σ schedule for L=174, baked in), decodes, and writes a stereo WAV. Flags: --L, --seed, --sample_rate, --output. The sampler uses its own RNG for the ping-pong noise, so output differs sample-wise run to run (expected for a stochastic sampler).

Music (Small-Music) — bonus bundle

The same pipeline runs Stable Audio 3 Small-Music (stabilityai/stable-audio-3-small-music), which is architecturally identical to SFX — same DiT/decoder config, only the weights differ. The music artifacts live under music/ with a longer latent length L = 388 (a 30 s prompt → 36 s canvas). Reuse the same `sa3_runner` and `lib/` from the repo root (the σ schedule is identical to SFX and baked in; --L 388 sizes the latent and constants).

music/
  sa3_cond_music.pte          # text encoder + conditioning (small-music weights)
  dit_music.pte               # diffusion denoiser           (QNN/HTP, FP16) — run 8x
  sa3_same_decoder_L388.pte   # latent -> audio decoder      (Conv1d->Conv2d, L=388)
  consts/                     # L=388 constants + tokenized "lo-fi hip hop beat, 90 BPM"

bash

adb shell mkdir -p /data/local/tmp/sa3/music
adb push music/sa3_cond_music.pte music/dit_music.pte \
         music/sa3_same_decoder_L388.pte /data/local/tmp/sa3/music/
adb push music/consts /data/local/tmp/sa3/music/

adb shell '
  cd /data/local/tmp/sa3
  LD_LIBRARY_PATH=$PWD/lib ADSP_LIBRARY_PATH=$PWD/lib \
    ./sa3_runner --cond music/sa3_cond_music.pte --dit music/dit_music.pte \
                 --decoder music/sa3_same_decoder_L388.pte --consts music/consts \
                 --L 388 --output music_out.wav --seed 0'

adb pull /data/local/tmp/sa3/music_out.wav .

Generates 36 s of stereo music in ~8 s on device (conditioning ~2.3 s, 8 DiT steps on the NPU ~1 s, decode ~4.5 s).

Why the QNN DiT works (the non-obvious part)

A first lowering ran fully on HTP and fast, but produced garbage (cos 0.36 vs FP32). It was not a precision problem — eager all-FP16 matched the golden at 0.99997. The bug was in the QNN lowering: the self-attention key-padding mask, built in-graph as a static constant (aten.full(bool)) concatenated with a bool input, is miscomputed on HTP. Fix: precompute that mask as a float graph input (padext.bin) and inject it, so the broken in-graph build never runs → 0.36 → 0.998 at full FP16 speed. Memory tokens, RoPE freqs, and the Fourier timestep table are externalized the same way.

License

—Model weights derive from Stability AI's Stable Audio 3 and are governed by the Stability AI Community License. Review it before any commercial or redistribution use; this repo is a research/educational port.
—`libqnn_executorch_backend.so` is part of ExecuTorch (BSD-3).
—Qualcomm QNN libraries are not included — obtain them under your own QNN SDK license terms.

Ported with Claude Code. Companion: the OmniVoice QNN port. Build host AOT-compiles on x86-64 Linux (QNN SDK ≥ 2.37); runner cross-compiled for Android arm64.