shangeth/expresso-mimi-codes

Expresso — Mimi Codes (k = 32)

Pre-extracted Kyutai Mimi tokens (all 32 codebooks) for both the read and conversational subsets of Expresso. Source audio + transcripts live in `shangeth/expresso`; this dataset publishes the discrete-token version for training Mimi-based speech models without re-extracting.

⚠️ License: CC-BY-NC-4.0 — non-commercial use only.

Why Expresso for Wren?

Expresso is the most directly relevant dataset for speech disentanglement research — the same speakers utter similar content across different expressive styles (happy, sad, confused, whispered, animal, etc.). Useful for studying:

• —Style vs content vs speaker identity
• —How expressive variation is encoded across Mimi's 32 codebooks (semantic + acoustic hierarchy)
• —Controllable style transfer at fixed speaker identity

Configs

• —`read` — ~11.6k mono utterances with human transcripts.
• —`conversational` — ~15.9k mono per-utterance turns from stereo dialogues, transcribed with Whisper Large V3 Turbo (3.0% overall WER on a held-out human-transcribed set).

Why 32 codebooks?

Most published speech-codec datasets use only the first 8 of Mimi's 32 codebooks (1 semantic + 7 acoustic), which is enough for the original Moshi recipe. We publish all 32 so you can:

• —Train models on more codebooks for higher resynthesis fidelity
• —Study which codebooks carry which content (style/timbre/prosody)
• —Slice codes[:k] at load time to use any prefix

k_codebooks is stored per row so the schema works for both k=32 and any subset you slice.

Schemas

read

conversational

Extraction details

• —Source audio: `shangeth/expresso` (the official Expresso tar, segmented and built into HF format)
• —Codec: `kyutai/mimi` @ 24 kHz, 12.5 fps, codebook size 2048 (fits int16)
• —Resampling: 48 kHz mono → 24 kHz before encoding
• —Conversational text: machine-transcribed (Whisper Large V3 Turbo with anti-hallucination decoding)

Usage

from datasets import load_dataset
import torch

# Pick a config
read = load_dataset("shangeth/expresso-mimi-codes", "read",          split="train")
conv = load_dataset("shangeth/expresso-mimi-codes", "conversational", split="train")

ex = read[0]
codes = torch.tensor(ex["codes"], dtype=torch.long)  # [32, n_frames]
print(f"speaker={ex['speaker_id']} style={ex['style']} | {ex['text'][:60]}")
print(f"codes shape: {codes.shape}  ({codes.shape[1]/12.5:.2f}s @ 12.5 fps)")

# Use only the first 8 codebooks (Moshi-style)
codes8 = codes[:8]

# Decode back to 24 kHz audio
from transformers import MimiModel
mimi = MimiModel.from_pretrained("kyutai/mimi").cuda().eval()
with torch.no_grad():
    wav = mimi.decode(codes.unsqueeze(0).cuda()).audio_values[0].cpu()  # [1, T] @ 24 kHz

Links

• —Audio dataset: `shangeth/expresso`
• —Extraction code: github.com/shangeth/wren-datasets
• —Wren research: github.com/shangeth/wren

Citation

@inproceedings{nguyen2023expresso,
  title     = {Expresso: A Benchmark and Analysis of Discrete Expressive Speech Resynthesis},
  author    = {Nguyen, Tu Anh and Hsu, Wei-Ning and D'Avirro, Antony and Shi, Bowen and
               Gat, Itai and Fazel-Zarani, Maryam and Remez, Tal and Copet, Jade and
               Synnaeve, Gabriel and Hassid, Michael and Kreuk, Felix and Adi, Yossi and Dupoux, Emmanuel},
  booktitle = {Interspeech},
  year      = {2023}
}

@misc{wren2026,
  title  = {Wren: A Family of Small Open-Weight Models for Unified Speech-Text Modelling},
  author = {Shangeth Rajaa},
  year   = {2026},
  url    = {https://github.com/shangeth/wren}
}

License

CC-BY-NC-4.0 — non-commercial use only. See `shangeth/expresso` original_metadata/LICENSE.txt.