litert-community/PANNs-CNN14-AudioSet-LiteRT

Measured on device (edge-compat): Galaxy S26 · LiteRT 2.2.0 · GPU (ML Drift) · 12.2 ms p50 (2026-08-26); Galaxy S26 · LiteRT 2.2.0 · NPU (QNN/HTP) · 3.89 ms p50 (2026-08-26); Raspberry Pi 5 · LiteRT 2.2.0.dev20260804 · CPU/XNNPACK, 4 threads · 391 ms p50 (2026-08-31); browser · Chromium 151 on M4 Max · LiteRT.js 2.5.3 · WebGPU · 13.2 ms p50 · output matches CPU (2026-08-11). Record: https://github.com/john-rocky/edge-compat/blob/main/cards/panns-cnn14-audioset/CARD.md

PANNs CNN14 — LiteRT (on-device AudioSet tagging, GPU CNN + host log-mel)

PANNs CNN14 (Cnn14_mAP=0.431) general sound-event tagging, converted to LiteRT with the CNN body running fully on the `CompiledModel` GPU (ML Drift) on Android. Given ~10 s of audio it predicts probabilities over the 527 [AudioSet](https://research.google.com/audioset/) classes — speech, music, instruments, animals, vehicles, alarms, household sounds, and so on. AudioSet tagging is multi-label: several tags can be high at once.

[image]

waveform[320000] (32 kHz) →[host: log-mel]→ logmel[1,1,1001,64] →[GPU: CNN14]→ probs[1,527] (sigmoid)

On-device (Pixel 8a, Tensor G3 — verified)

How it converts (litert-torch) — and why the log-mel is host-side

PANNs builds its spectrogram with torchlibrosa, whose STFT is a DFT-as-Conv1d — so there is no FFT op and the whole raw-audio→tags graph is almost GPU-clean; the only blocker is the STFT centering reflect-pad (one GATHER_ND, removable via pad_mode='constant', corr 1.0). But the converted spectral front-end is unusable: litert-torch lowers the giant 1024-tap DFT-conv incorrectly (fp32 tflite corr ≈ 0.19), and the power spectrum |STFT|² (~1e6) overflows fp16 on Mali → NaN.

So the spectral front-end is computed on the CPU (the Whisper/Kokoro pattern), matched to torchlibrosa exactly, and only the CNN body rides the GPU:

• —log-mel (host) — reflect-pad center, periodic Hann, 1024-pt FFT, power, mel matmul (librosa.filters.mel, slaney), 10·log10(max(mel,1e-10)). Validated host-vs-torch corr 1.000000 (max|d| 0.0017). The mel basis is shipped here as mel_basis.bin [64, 513].
• —CNN14 body (GPU) — bn0 + 6 conv blocks + mean/max time-pool + 2 FC + sigmoid. Pure CNN, converts at corr 1.000000 in fp32 and fp16, op-check banned NONE / >4D 0, one delegatable graph.

Files

Preprocessing

Mono 32 kHz, padded/truncated to 10 s (320000 samples), values in [-1, 1]. Compute the log-mel as above → [1,1,1001,64]. The output 527 sigmoid probabilities are per-class (multi-label); take the top-K as tags.

Minimal usage

Android (Kotlin, CompiledModel GPU)

// staged into filesDir by an install script (162 MB — too big for assets)
val model = CompiledModel.create(File(ctx.filesDir, "cnn14_audioset_fp16.tflite").absolutePath,
    CompiledModel.Options(Accelerator.GPU), null)
val inputs = model.createInputBuffers(); val outputs = model.createOutputBuffers()
inputs[0].writeFloat(logmel)          // [1,1,1001,64] host log-mel (see Python below)
model.run(inputs, outputs)
val probs = outputs[0].readFloat()    // [527] sigmoid, multi-label -> top-K tags

Python (desktop verification)

import numpy as np, soundfile as sf
from ai_edge_litert.interpreter import Interpreter

SR, NFFT, HOP, NMEL, CLIP = 32000, 1024, 320, 64, 320000
wav, _ = sf.read("clip_32k.wav", dtype="float32")               # mono 32 kHz
x = np.zeros(CLIP, np.float32); n = min(len(wav), CLIP); x[:n] = wav[:n]

# torchlibrosa-exact log-mel: center reflect-pad, periodic Hann, |rFFT|^2, mel, 10*log10
pad = np.pad(x, NFFT // 2, mode="reflect")
win = 0.5 - 0.5 * np.cos(2 * np.pi * np.arange(NFFT) / NFFT)
frames = 1 + CLIP // HOP                                        # 1001
power = np.stack([np.abs(np.fft.rfft(pad[t*HOP:t*HOP+NFFT] * win))**2 for t in range(frames)])
fb = np.fromfile("mel_basis.bin", np.float32).reshape(NMEL, 513)
logmel = (10.0 * np.log10(np.maximum(power @ fb.T, 1e-10))).astype(np.float32)

it = Interpreter(model_path="cnn14_audioset_fp16.tflite"); it.allocate_tensors()
it.set_tensor(it.get_input_details()[0]["index"], logmel[None, None]); it.invoke()
probs = it.get_tensor(it.get_output_details()[0]["index"])[0]   # [527]

labels = open("audioset_labels.txt").read().splitlines()
for i in probs.argsort()[::-1][:5]:
    print(f"{probs[i]:.3f}  {labels[i]}")

Performance

Measured on a Pixel 8a (Tensor G3, Android 16) with the standard TFLite `benchmark_model` tool — 10 warm-up runs then 50 timed runs, reported as the tool's mean.

Any on-device figure recorded when this model shipped came from a different runtime. It was taken through LiteRT's own CompiledModel accelerator (logcat reports it as LITERT_CL), which is the path the Kotlin sample app and the LiteRT API use, and it appears elsewhere on this card. The rows above are the classic TFLite OpenCL delegate, measured with a tool anyone can download and re-run. The two are not comparable, so read the rows above as a reproducible floor rather than as this model's speed on LiteRT.

XNNPACK declines these fp16 graphs — it reports failed to delegate DEPTHWISE_CONV_2D and then fails to allocate tensors — so there is no usable CPU number. Disabling XNNPACK falls back to reference kernels, which measured about 20× slower than the GPU on models of this size and would not represent CPU inference anyone would ship.

Snapdragon NPU (Hexagon)

The NPU is 3.13x faster than the GPU (3.89 ms against 12.15 ms) and loads 6.05x faster (158 ms against 957 ms).

Measured on a Samsung Galaxy S26 (Snapdragon 8 Elite Gen 5 / SM8850, Hexagon v81, Android 16) with LiteRT CompiledModel 2.2.0, one accelerator per process, 5 warm-up runs then N=50 timed runs, median reported. Every run held thermal status NONE throughout. Headroom 0.66–0.67, where 1.0 is the throttling threshold.

The NPU rows ran the published file unchanged. LiteRT compiled it for the Hexagon on the device at first load. That first compile took 2.7 s here. The load column above is the cached load every later run pays. Recipe and the runtime libraries it needs: NPU guide.

GPU wiring: GPU guide.

Raspberry Pi 5 (CPU)

Measured on a Raspberry Pi 5 Model B Rev 1.1 (8 GB, Raspberry Pi OS 64-bit) with the LiteRT `benchmark_model` tool from litert-cli-nightly 0.2.0.dev20260805: CPU inference (XNNPACK, 4 threads), 3 invocations per file of 10 warm-up plus 50 timed runs (the tool caps a phase at 150 s, so very slow graphs run fewer — the Runs column is the actual timed total). The latency is the median across invocations; the spread is the min–max over all timed runs. No thermal throttling occurred during these runs (vcgencmd get_throttled stayed 0x0).

License

Code Apache-2.0; weights Cnn14_mAP=0.431.pth CC-BY-4.0 (Zenodo). AudioSet ontology © Google, CC-BY-4.0. Upstream: qiuqiangkong/audioset_tagging_cnn.

Citation

@article{kong2020panns,
  title={PANNs: Large-Scale Pretrained Audio Neural Networks for Audio Pattern Recognition},
  author={Kong, Qiuqiang and Cao, Yin and Iqbal, Turab and Wang, Yuxuan and Wang, Wenwu and Plumbley, Mark D},
  journal={IEEE/ACM Transactions on Audio, Speech, and Language Processing},
  year={2020}
}