Strg-Alt-Entf-0x00/FireRedVAD-ESP32-P4

FireRedVAD Models for ESP32-P4

Converted FireRedVAD models optimized for ESP32-P4 microcontrollers. All models use the custom .frvd binary format with native RISC-V PIE (Position Independent Execution) acceleration.

Source Code: https://github.com/Strg-Alt-Entf-0x00/firered-vad-esp32-p4

Quick Start

cd examples/console_vad
pip install huggingface-hub
python download_models.py

idf.py build flash monitor

firevad> vad_model_list
firevad> vad_model_load stream-vad/int8-ch/firered-stream-vad-int8-ch.frvd
firevad> vad_infer_mic

Available Models

stream-vad/ — Real-time streaming (causal, 10ms latency)

Best for real-time voice activity detection. Model is fully causal — no future context. Runs in ~4.5ms per 10ms frame on ESP32-P4 @ 400MHz.

vad/ — Offline batch VAD (non-causal, 1-second chunks)

Uses bidirectional context. Higher accuracy than stream-vad, but adds latency. Not suitable for real-time streaming.

aed/ — Audio Event Detection (Speech / Music / Singing)

Multi-class audio classifier. Identifies speech, music, and singing simultaneously.

Quantization Explained

Why INT8-CH (Per-Channel) is Recommended

Standard per-tensor INT8 quantization assigns one global scale factor per weight matrix. DFSMN architectures have wide variance in weight distribution across output channels — a single scale factor cannot capture this range accurately, causing silent accuracy loss.

Per-Channel INT8 (`int8-ch`, Version 4 in the `.frvd` format) assigns one scale factor per output channel. This preserves near-FP32 accuracy at INT8 speed and memory cost.

Real-time budget for 10ms frames: 10ms. Anything above 10ms (>100% load) causes audio drops.

🎤 Audio Frontend & AGC

We achieved a highly robust audio frontend using the INMP441 I2S microphone combined with a custom Automatic Gain Control (AGC) and DC-Offset High-Pass filter:

• —Dynamic Range Compression: Without AGC, voice volume drops steeply from -27 dB (at 10cm) to -47 dB (at 2m). With AGC enabled, the volume stays rock-solid between -30 dB and -37 dB!
• —Distance Boost: The AGC automatically boosts weak signals at a 2-meter distance by +10 dB, while seamlessly throttling loud, up-close speech.
• —SNR: Crystal clear 23 dB SNR.
• —Golden Test: Bit-parity and VAD-quality mathematically verified against original PyTorch outputs.

Hardware Requirements

• —MCU: ESP32-P4 (RISC-V dual-core, 400MHz)
• —PSRAM: 32 MB
• —Flash: 16–32 MB
• —RAM at runtime: ~150 KB
• —Microphone: INMP441 or equivalent I2S digital microphone @ 16kHz

Note: INT8 and INT8-CH models use ESP32-P4 PIE vector instructions (esp.vmulas.s8.xacc etc.) with mandatory 16-byte memory alignment, handled automatically by the runtime. FP32/INT16 models work on other ESP32 variants (S2, S3) but without PIE acceleration.

Known Limitations (Honest Assessment)

1. 1.Noise sensitivity: Performance degrades in low-SNR environments (loud machinery, strong wind). False positive rate increases at SNR < 5dB.
2. 2.Microphone dependency: Model was trained on clean 16kHz PCM. A high-quality I2S microphone with hardware PGA gain control is required for reliable results.
3. 3.No built-in noise suppression: The ESP-IDF runtime does not include NS/AEC. Echo cancellation is available via the shared APLL (I2S0 + I2S1 synchronized clocking).
4. 4.APLL sharing warning: When both TX and RX I2S ports are active, the ESP32-P4 APLL runs at 8,191,999 Hz instead of 8,192,000 Hz (1 Hz deviation). This is hardware-expected behavior, not a bug. Both ports share the same clock, which is ideal for AEC.

.frvd File Format

Custom binary format, version-tagged in header byte [4..7]:

Header (32 bytes):
  [0..3]   Magic: "FRVD"
  [4..7]   Version: 1=fp32, 2=int8, 3=int16, 4=int8-per-channel
  [8..11]  Model type: 0=VAD, 1=Stream-VAD, 2=AED
  [12..15] Total parameter count
  [16..23] DFSMN block count + DNN layer count
  [24..31] Reserved

Architecture Metadata (32 bytes):
  Input dim, hidden size, projection size, output dim,
  lookback order/stride, lookahead order/stride

CMVN block:
  dim (uint32) + means[dim] (float32) + istd[dim] (float32)

Layer data (sequential):
  Per tensor: CRC32 name hash + element count + [scale per channel for int8-ch] + data

Conversion Pipeline

Original FireRedVAD PyTorch checkpoints -> .frvd:

# Requirements
pip install torch kaldiio numpy

# Stream-VAD INT8-CH (recommended)
python tools/converter/export_weights.py \
    --model-dir tools/original_models/Stream-VAD \
    --output-dir examples/console_vad/converted_models/stream-vad/int8-ch \
    --model-type stream-vad \
    --quantize-int8-per-ch

# Stream-VAD INT8
python tools/converter/export_weights.py \
    --model-dir tools/original_models/Stream-VAD \
    --output-dir examples/console_vad/converted_models/stream-vad/int8 \
    --model-type stream-vad \
    --quantize-int8

# Verify conversion
python tools/converter/verify_conversion.py \
    --frvd examples/console_vad/converted_models/stream-vad/int8-ch/firered-stream-vad-int8-ch.frvd

License & Attribution

Original Models

• —FireRedVAD by Xiaohongshu (FireRedTeam) — Apache 2.0
• —Source: https://github.com/FireRedTeam/FireRedVAD
• —HuggingFace: https://huggingface.co/FireRedTeam/FireRedVAD

ESP32-P4 Port

• —FireRedVAD-ESP32-P4 by Strg-Alt-Entf-0x00 — Apache 2.0
• —Repository: https://github.com/Strg-Alt-Entf-0x00/firered-vad-esp32-p4

Citation

@misc{fireredvad-esp32p4,
  title={FireRedVAD for ESP32-P4: Optimized Voice Activity Detection for Embedded Systems},
  author={Strg-Alt-Entf-0x00},
  year={2026},
  howpublished={\url{https://github.com/Strg-Alt-Entf-0x00/firered-vad-esp32-p4}},
}