mlr2000/vocoder-small-speaker-encoder

VocBulwark speaker encoder

Standalone speaker encoder: turns a raw waveform into a fixed 768-dimensional speaker embedding. It is the conditioning front-end for the companion VocBulwark vocoder — compute an embedding once from a reference clip of a speaker, then pass it to the vocoder to synthesize in that voice. The embedding is also usable on its own for speaker verification / similarity.

Wav2Vec2-based encoder with attentive statistics pooling, trained with a GE2E objective. Self-contained: loads with trust_remote_code=True, no training repo required.

Companion Models

This speaker encoder is part of a set of 6 repositories:

Usage

import torchaudio, torchaudio.functional as AF
from transformers import AutoModel

enc = AutoModel.from_pretrained("mlr2000/vocoder-small-speaker-encoder", trust_remote_code=True).eval()

wav, sr = torchaudio.load("reference.wav")             # [C, T]
wav = wav.mean(0, keepdim=True)                        # mono [1, T]
if sr != enc.config.raw_sample_rate:                   # encoder expects 22.05 kHz
    wav = AF.resample(wav, sr, enc.config.raw_sample_rate)

emb = enc.embed(wav)          # [1, 768]  — feed as speaker_embedding to the vocoder

See `example_roundtrip.ipynb` in this repo for the full pipeline (reference clip → embedding → vocode → detect watermark).

Notes

• —Input: mono waveform at 22.05 kHz (config.raw_sample_rate). Resample first if your audio differs.
• —Output: [B, 768] L2-comparable speaker embeddings.
• —Pair with the VocBulwark small vocoder trained jointly with this encoder. An embedding from a different encoder will not condition the vocoder correctly.
• —This encoder is paired with mlr2000/vocoder-small. For the large vocoder (mlr2000/vocoder-large) use the companion large speaker encoder (mlr2000/vocoder-large-speaker-encoder).
• —Not intended for speaker identification or surveillance of individuals without consent.

Citation

If you use this model, please cite:

@misc{muletta2026,
  title  = {Training a Discriminator-Free Foundation Vocoder 
             with Integrated Audio Watermarking},
  author = {Muletta, Romolo and Deriu, Jan},
  year   = {2026},
  note   = {VT2 Project Report, ZHAW School of Engineering}
}

License

cc-by-4.0. Trained on MLS (CC-BY-4.0) and Common Voice (CC0). Builds on BigVGAN (MIT) and wav2vec 2.0 (Apache-2.0). Please retain attribution when redistributing or building on this model.