litert-community/sopro-v2-turbo

Sopro v2 turbo on LiteRT (CompiledModel)

[image]

Sample from the converted model (plain fp16 weights + int8 AR, streaming vocoder) cloning a CC0 volunteer voice from kyutai/tts-voices (voice-donations/0a67): assets/hero_sample.wav — “The morning light fell across the garden while a small bird sang beside the open window.” Source and license of the reference voice: assets/ATTRIBUTION.md.

What it is

These classic .tflite graphs convert Samuel Vitorino's Sopro v2 turbo, a reference-conditioned text-to-speech model for English, European Portuguese, French and German, to the LiteRT CompiledModel API. The package provides fp32 reference graphs, plain weight-only fp16 graphs and a native-int8 merged AR alternative. It produces 24 kHz audio and includes a streaming vocoder, while resampling, spectral DSP, token sampling and the acoustic solver loop stay on the host. The upstream model and source are Apache-2.0; the included LICENSE is retained verbatim. Conversion used sopro==2.2.0 and upstream model revision f747f9edfb7b0233a3b7105af3a75603a7213d26.

Files

The file sizes below are exact bytes. There are fourteen fp32 graph files, fourteen plain wfp16 graph files and one native-int8 merged AR file. The Android variants add eight fp32, seven wfp16 and one int8 graph file (rows marked "Android variant"), plus the app source under android/. Separate AR prefill/step files and merged alternatives are included so callers can choose one form without loading both. This table covers the executable artifacts, host assets and conversion sources.

The wfp16/ files use plain FLOAT_CASTING: the final vocoder head is also stored as fp16, with float32 public I/O and CPU computation. The native-int8 AR uses per-channel int8 weights with dynamic activation quantization inside its CPU kernels. No speaker, acoustic or vocoder int8 candidate is included.

Raw host bins are little-endian; host_assets/host_assets.json gives each tensor's shape, dtype, byte offset and byte size. The measured pipeline used the original fp32 NPZ embedding tables. The requested fp16 raw embedding tables are a storage option, not an independently quality-gated host change. The original NPZs remain available for the validated path.

Graph contract

Batch size is one. Float inputs and outputs are float32 for every weight variant; token IDs, gather indices and last_index are int32. Bind actual signature names and buffer order from contract.json, not inferred tensor-index order. The ordinary signature is serving_default; merged AR exposes prefill and step with the same corresponding I/O contracts.

Acoustic (T,N) is (2048,512) or (4096,1024). The _t4096 pair is the second bucket. The semantic encoder follows the installed package's 500-position crop before its fixed interpolation to 235 tokens. The fixed prefix uses eight style vectors, 120 prompt tokens and BOS, leaving at most 127 text tokens in the 256-position bucket; reject or segment a longer prefix before inference. Source activation functions are retained: semantic/vocoder erf-GELU and the acoustic DiT's native tanh-GELU.

The Android variants (_r6 / _r9 files) keep these computations and change only inputs the GPU delegate could not lower: AR prefill takes last_onehot [1,256] instead of last_index; acoustic condition takes token_onehot [1,N,4377] and frame_onehot [T,N] instead of semantic_tokens and frame_to_token; the semantic encoder returns digit_logits [1,235,27] and the host takes the argmax; the style prefix is shipped as fp32. Their signatures are listed in contract.json under android_contract_deltas, and their Mac CPU outputs equal the originals (maximum difference 0.0).

Host-side steps

• —Load mono audio, resample and apply the upstream reference normalization. Compute separate speaker, Whisper and acoustic mel frontends, including the speaker frame LayerNorm and Whisper's zero append, logarithmic floor and dynamic-range clamp. Normalize the acoustic reference mel with the supplied mean and standard deviation.
• —Tokenize with the language tag, gather text and semantic embedding tables, build the style/text/prompt/BOS prefix, and create RoPE tables and runtime additive masks. Keep packed K/V on the host; each step returns only its new slice. Attention includes the current key/value and masks invalid cache slots with −10,000.
• —Sample at temperature 0.8, top-k 25 and top-p 0.9, using the probability-below-kth rule, shifted top-p removal and 1e-8 normalization clamps. BOS is always masked; EOS is enabled from prediction ten. Use one uniform draw and inverse CDF. The evaluated free-running seed rule was NumPy default_rng(2000+i).
• —Construct frame-to-token indices from the actual token/frame lengths. Run the two Euler steps on the sway grid [0,0.29289323,1], using the same initial noise when measuring parity, and re-mask the reference prompt after each step with sigma_min 1e-6.
• —Denormalize the solved mel, retain 32 reference-context frames before generated frames, run the vocoder, split its head output into log-magnitude and phase, apply exp/clamp and complex phase, and perform host iSTFT with FFT 1024, hop 256 and periodic Hann. Overlap-add, window-envelope division and center cropping remain host operations.
• —Remove reference-context samples and apply output_gain, leading/trailing trim, single-segment join, soft_limit, then an 80 ms fade-out without fade-in. Streaming iSTFT state is ola, env, tail_start, processed_frames, and emitted_samples.

fp32 FFT contract: the acoustic mel mirror explicitly selects NumPy's fp32 FFT path with an exact power-of-two normalization and scale restoration. A mathematically equivalent implementation that dispatches to fp64 changes fp32 rounding. Preserve the implementation in conversion/host_dsp.py when reproducing these measurements; another FFT library needs its own parity check.

Reference contract

The static host input is exactly 10 seconds at 24 kHz, or 240,000 samples, resampled to 160,000 samples at 16 kHz for the encoders. Longer references are cropped; deterministic zero-padding of shorter inputs is the host's fixed-shape option. Short-reference padding was not evaluated: both measured references were at least ten seconds, and normalization preceded cropping. This differs from upstream crop_on_pause, which can append random room tone and is bypassed here. Upstream describes voice cloning from 5–20 seconds of reference audio; that guidance is not a measurement of this fixed bucket.

Buckets and limits

Streaming start emits 37 feature frames, ordinary steps emit 64, and flush emits 27. For a partial final chunk, the host replays the last 128 real mel frames through a separate start/step, keeps only the remainder from that step, then flushes that replay state. This is the measured exact tail strategy; it does not cover arbitrarily short inputs. The acoustic solver itself is still offline.

The long fixtures generated 16.416–17.824 seconds of raw audio, using 385–418 tokens and at most 635 cache positions. All four used the long bucket and streaming vocoder. These are measured long-utterance tests, not thirty-second saturation tests; the thirty-second bound is established separately by the static capacity arithmetic.

Android

The android/ directory holds a Kotlin sample app (Compose, LiteRT CompiledModel) that runs the whole pipeline on the phone: the three mel front-ends with an fp32 FFT, the 24→16 kHz resampler, the SentencePiece tokenizer with the language tag, the packed-KV AR loop with the sampler, the two-step acoustic solver, the streaming vocoder with a host iSTFT, and the post-processing chain, then plays the audio through AudioTrack while it is being decoded. Every host step was gated on the JVM against the Python dumps before the device (worst tensor difference 2e-5 on the mels, 2.4e-7 on the iSTFT, sampler picks 2,920/2,920, tokenizer 1,214/1,214) and again on the phone, where the device chains reproduce the Mac results (fp32 teacher-forced raw waveform corr ≥ 0.9999993, 24/24).

Five graphs were re-exported without any numerical change so that the GPU delegate accepts them; the new files carry the _r6/_r9 suffix in their directories and supersede the originals on Android (Mac CPU parity against the originals: maximum difference 0.0). sopro_ar_merged: the prefill's last-row gather became a one-hot float input last_onehot [1,256]. sopro_acoustic_condition: the token-embedding lookup and the frame gather became one-hot float selections (token_onehot [1,N,4377], frame_onehot [T,N]). sopro_acoustic_velocity: the BROADCAST_TO lowering was removed. sopro_semantic_encoder: the graph returns the FSQ digit logits [1,235,27] and the host takes the argmax and forms the token (bases 1, 7, 35, 175, 875). sopro_style_prefix: shipped as fp32 (3.2 MB). Signatures and buffer order are in contract.json.

Measured placement on a Galaxy S26 (SM-S942Q, Adreno), LiteRT 2.2.0, release build, screen on, the phone held by the test alone, battery 39–43 °C. Warm latency is the synchronized run + read time of one call, median over the streaming suite.

Streaming on that placement, 36 utterances (the two private references and the CC0 demo voice × the 12 test sentences), release build:

Time to first audio is the tap-to-first-PCM-chunk time; the AR loop and the two velocity calls precede the first vocoder chunk. Quality on the shipped placement: Whisper-turbo WER 0.56 % / 3.70 % / 0 % and speaker cosine 0.939 / 0.916 / 0.777 for reference one / reference two / the demo voice (the demo voice's own PyTorch baseline is 0.781; its clip is a plain volunteer recording). Playback applies the output gain and soft limit per chunk; the saved WAV goes through the full offline post-processing.

Pixel 8a (Mali): not measured yet.

What is not included

Mac CPU numbers and Galaxy S26 Android numbers are included; the Pixel 8a (Mali) column is not measured yet. No streaming acoustic graphs, NPU verification or private reference voices are included. A macOS ai-edge-litert 2.2.0 GPU-only CompiledModel ends the process with SIGSEGV on a trivial Linear-plus-ReLU graph at creation; the 2.1.6 wheel runs the same file on Metal, so no Mac GPU numbers are given here. Private LibriSpeech reference readers and their generated audio are excluded from this package.

Gates and results

All execution measurements in this section used Apple M4 Max CPU, four threads, ai-edge-litert 2.2.0 CompiledModel, with float32 public tensors and the weight dtype named by the table. Source checks used PyTorch 2.11.0 / torchaudio 2.11.0 / Sopro 2.2.0 in float32. The machine was shared; timings are contended. The short suite contains two private reference voices and twelve sentences, giving 24 utterances in English, Portuguese, French and German. Four additional long utterances cover English and German. This is a small conversion-gate suite, not broad model-quality coverage.

For fp32 per-graph floats, the measured rule is finite values, correlation at least 0.9999, norm ratio within 1±0.001, and max error at most max(0.001,0.0001×reference_absmax) on valid positions. Integer output agreement is checked separately. Reduced-precision acoustic graphs are judged in solved-mel and output log-mel space; vocoders are judged in waveform space. Absolute phase-channel tensor errors do not determine audible parity.

fp32 reference set

The source's own fp32 streaming/offline difference reached 2.15493e-4; its fp64 waveform check reached 4.96575e-13. This source summation-order difference was accepted without changing the graphs.

Plain wfp16 set

The full reduced chain uses the wfp16 neural stages and native-int8 merged AR, with original fp32 host tables. Teacher forcing fixes the reference/generated token sequence and initial noise, so AR precision does not select a different downstream sequence in that test. The all-wfp16 AR alternative was also executed on the same teacher path. Its minimum raw-waveform correlation was 0.977468237158 on one Portuguese utterance; that value is informational under the acoustic mel-domain rule, not omitted or treated as waveform equivalence. HNR uses 25 ms autocorrelation frames with 10 ms hop and the fp32 output's HNR>0 dB voiced mask; energy ratio compares 4–12 kHz with 0–4 kHz.

Native-int8 merged AR

WER and speaker similarity

ASR: openai-whisper turbo / large-v3-turbo, package 20250625, float32 CPU, four threads; language forced per utterance, 16 kHz float32 array input, no ffmpeg. WER uses jiwer 4.0.0, Whisper EnglishTextNormalizer for English and BasicTextNormalizer for Portuguese/French/German. Values are the unweighted mean of per-utterance WER, in percent. Speaker cosine uses the source float32 speaker encoder on each output and its reference embedding. Hardware/runtime and contended conditions are those stated above; these are quality comparisons, not speed comparisons.

All 24 reduced-pipeline final waveforms were finite, with RMS above 0.001 and peak at most 1.0; measured durations were 4.4693–6.0053 seconds. The quality gate required mean speaker cosine no more than 0.03 below the oracle, minimum cosine at least 0.80, and mean WER no more than three percentage points above the oracle.

Contended CPU timing

Apple M4 Max CPU; ai-edge-litert 2.2.0 CompiledModel; four threads; contended. The graph table includes input-buffer writes, synchronous execution and output-buffer reads, with two warmups followed by eight timed calls on recorded real inputs. Each row names its weight format; public I/O is float32 or the declared int32 indices. Model load is excluded. Other workloads shared the machine, so these are observed latencies, not latency guarantees or phone estimates.

The reduced free-running pipeline, including reference processing, AR, acoustic solving, streaming vocoder and host DSP/post-processing, had RTF 0.442386 median and 0.600927 maximum over 24 utterances under the same contended four-thread CPU conditions. Model loading and audio-file serialization were excluded. RTF is pipeline wall seconds divided by final audio seconds. The timed configuration uses native-int8 merged AR, wfp16 remaining graphs and the original fp32 host tables.

Minimal usage

These examples call only the speaker graph. The Python block uses zero-valued host mel as a runnable tensor-I/O smoke input; replace it with the normalized speaker log-mel [1,80,1001] from your own reference frontend for real use. Run from the repository root after installing the pinned environment in REPRODUCE.md.

from pathlib import Path
import numpy as np
from ai_edge_litert.compiled_model import (
    CompiledModel, Options, CpuOptions, HardwareAccelerator,
)

path = Path("fp32/sopro_speaker_encoder_fp32.tflite")
model = CompiledModel.from_file(str(path), options=Options(
    hardware_accelerators=HardwareAccelerator.CPU,
    cpu_options=CpuOptions(num_threads=4),
))
inputs = model.create_input_buffers(0)
outputs = model.create_output_buffers(0)
speaker_mel = np.zeros((1, 80, 1001), dtype=np.float32)
try:
    inputs[0].write(speaker_mel.ravel())  # args_0
    model.run_by_index(0, inputs, outputs)
    id_emb = np.array(outputs[0].read(192, np.float32)).reshape(1, 192)
    print(id_emb.shape, float(np.linalg.norm(id_emb)))  # output_0
finally:
    for buffer in inputs + outputs:
        buffer.destroy()
    model.close()

The Kotlin example follows the LiteRT Kotlin CompiledModel/TensorBuffer API shape. It is a design example, not an Android execution result; modelPath points to the downloaded fp32 speaker file, and hostMel is channel-major float32 speaker log-mel.

import com.google.ai.edge.litert.Accelerator
import com.google.ai.edge.litert.CompiledModel

fun speakerEmbedding(modelPath: String, hostMel: FloatArray): FloatArray {
    require(hostMel.size == 80 * 1001)
    val model = CompiledModel.create(
        modelPath, CompiledModel.Options(Accelerator.CPU), null
    )
    val inputs = model.createInputBuffers()
    val outputs = model.createOutputBuffers()
    try {
        inputs[0].writeFloat(hostMel) // args_0: [1,80,1001]
        model.run(inputs, outputs)
        return outputs[0].readFloat().also { // output_0: id_emb
            require(it.size == 192)
        }
    } finally {
        inputs.forEach { it.close() }
        outputs.forEach { it.close() }
        model.close()
    }
}

Text examples and use

These plain-word sentences were used in the private test suite and contain no personal names or trademarks:

• —English: “The morning light fell across the garden while a small bird sang beside the open window.”
• —Portuguese: “A luz da manhã entrou pela janela e iluminou as flores que estavam sobre a mesa da cozinha.”
• —French: “Après la pluie, nous avons suivi le petit chemin qui descend vers le lac au milieu des arbres.”
• —German: “Nach dem Regen gingen wir den schmalen Weg entlang und hörten den Wind in den hohen Bäumen.”

Upstream disclaimer: “Please use the model for good: do not impersonate people.”

Conversion and verification code is included under conversion/. Supply your own authorized reference audio to reproduce speaker or full-pipeline checks. No private fixtures, reference audio, transcripts, oracle dumps or generated samples are included.