CoolFace
Datasetpublic

Trelis/libritts-bpe-tokens

libritts-bpe-tokens To learn about Trelis Enterprise Voice Services, see Trelis.com/voice-ai-services. GPT-2 BPE tokens of LibriTTS-R text_normalized transcripts. Each utterance is terminated with the EOS token (50256). Tokens are in column token_ids (list[int]), vocab=50,257. Splits Mirrors the source LibriTTS-R splits (filtered by parler-tts; total ≈ 538 h): split utterances hours train.clean.100 ~32 k ~53 h train.clean.360 ~112 k ~218 h… See the full description on the dataset page: https://huggingface.co/datasets/Trelis/libritts-bpe-tokens.

sourceHugging Facecc-by-4.0updated 4mo agoView on Hugging Face
0likes93downloads
Dataset Card

libritts-bpe-tokens

To learn about Trelis Enterprise Voice Services, see Trelis.com/voice-ai-services.

GPT-2 BPE tokens of LibriTTS-R text_normalized transcripts. Each utterance is terminated with the EOS token (50256). Tokens are in column token_ids (list[int]), vocab=50,257.

Splits

Mirrors the source LibriTTS-R splits (filtered by parler-tts; total ≈ 538 h):

splitutteranceshours
train.clean.100~32 k~53 h
train.clean.360~112 k~218 h
train.other.500~250 k~258 h
dev.clean~5.6 k~9 h

Source: `parler-tts/libritts_r_filtered`. Disjointness between splits is structural (HF split definition — no speaker overlap between train. and dev.).

Schema (one row per utterance)

columntypedescription
idstringSource utterance id (LibriTTS speaker_chapter_segment)
speakerstringLibriTTS speaker id
textstringtext_normalized from source
token_idslist[uint32]GPT-2 BPE tokens + EOS, vocab 50,257

Loading

python
from datasets import load_dataset

# Load the whole dataset (all 4 splits)
ds = load_dataset("Trelis/libritts-bpe-tokens")

# Just one split
clean_360 = load_dataset("Trelis/libritts-bpe-tokens", split="train.clean.360")

# Combine all train splits
all_train = load_dataset("Trelis/libritts-bpe-tokens",
                         split="train.clean.100+train.clean.360+train.other.500")

print(all_train[0])

Companion datasets (same audio content, different tokenization)

  • —Trelis/libritts-mimi-tokens
  • —Trelis/libritts-snac-tokens
  • —Trelis/libritts-neucodec-tokens

Reproducing

Encoder code: TrelisResearch/audio-bits (see scripts/encode_codec.py and scripts/tokenize_text.py).

Audio prep: source LibriTTS-R is 24 kHz. NeuCodec (16 kHz) is downsampled with torchaudio.functional.resample(method="sinc_interp_kaiser") (polyphase, anti-aliased). Mimi and SNAC consume 24 kHz directly. Utterances > 20 s are truncated to 20 s of audio (text is kept full).

Per-row token count = min(ceil(n_audio_samples * T / max_padded_samples), T) where T is the codec's per-batch output length, so the count tracks the model's own time grid and is exact for fixed-stride encoders.

License

CC-BY-4.0 (matches LibriTTS-R source). GPT-2 tokenizer (MIT).