aiacademy-kg/audio-3-speaker-dataset-v2

Three-Speaker Audio Dataset with Timbral Speaker Embeddings

A teaching dataset maintained by AI-Academy. It pairs single-speaker English utterances with precomputed timbral speaker embeddings, and is intended for coursework and exercises rather than for benchmarking or production systems.

The dataset deliberately contains one injected inconsistency; locating it is one of the intended exercises (see The injected anomaly).

Overview

Dataset structure

Each record has four fields:

Records are ordered by speaker and each shard contains exactly one speaker, so index ranges and shard boundaries coincide:

This layout is convenient for selective loading but has consequences for how the data may be partitioned; see the caveats below.

Speakers and provenance

The mixture of one natural and two synthetic voices is intentional: it allows the same corpus to serve both speaker-modelling exercises and exercises that contrast recorded speech with contemporary synthesis.

Speaker embeddings

The wavlm_embedding field was produced with `Orange/Speaker-wavLM-tbr` (the W-TBR variant), a WavLM-large derivative that encodes the timbral traits of a voice while suppressing prosodic ones. The procedure was:

1. 1.decode the audio to 16 kHz, averaging channels to mono;
2. 2.truncate the signal to the first 320,000 samples (20 seconds);
3. 3.apply mean-variance normalisation and the model's statistics-pooling head.

The output is a unit-norm 128-dimensional vector, so the cosine similarity between two embeddings reduces to their dot product. The authors of the model report an equal error rate of 1.68 % on the VoxCeleb1-clean verification task at a decision threshold of 0.472, which gives a principled reference point for exercises that require one.

Two consequences are worth stating explicitly. First, because of step 2, the embedding of any utterance longer than 20 seconds describes only its first 20 seconds. Second, the pipeline is reproducible: recomputing embeddings from the stored audio recovers the released vectors to a cosine similarity of approximately 0.998 or above, which makes reproduction itself a usable laboratory exercise.

Intended educational use

The dataset was assembled to support, among others, the following exercises.

Vector operations and similarity. The embeddings are unit-norm, low-dimensional and cleanly separated by speaker, which makes them a suitable substrate for a first treatment of cosine similarity and cosine distance, Euclidean distance on the unit sphere and its relation to the cosine, class centroids and centroid-based assignment, and the effect of normalisation on each of these.

Speaker verification. Constructing same-speaker and different-speaker trial pairs from the released vectors permits a full verification pipeline: score distributions, receiver operating characteristic and detection error tradeoff curves, equal error rate, and the calibration of a decision threshold, which may then be compared against the published value of 0.472.

Classification and representation analysis. Speaker identity is close to linearly separable in embedding space, so the corpus supports both simple classifiers and the diagnostic tooling around them: confusion matrices, learning curves, and projection by principal component analysis, t-SNE or UMAP.

Anomaly and label-noise detection. See the following section.

Signal processing. The stored files retain heterogeneous native sampling rates, so resampling, mono conversion, duration statistics, silence trimming and the extraction of spectral features are all exercisable on the raw audio rather than on prepared arrays.

Natural versus synthetic speech. One speaker is a recorded human and two are synthesised, which supports introductory work on synthetic-speech detection and on the acoustic differences between the two sources.

Speech recognition. The paired text field permits evaluation or fine-tuning of recognition models, with the caveat on transcript fidelity noted below.

Data partitioning. The corpus ships as a single train split. Designing a defensible partition — and recognising why the naive sequential one fails here — is itself part of the intended work.

The injected anomaly

One utterance in the corpus has been deliberately corrupted. Its audio was replaced with a recording of a different speaker who appears nowhere else in the dataset, while its speaker_id, its text and its position in the corpus were left untouched. The wavlm_embedding of that record was recomputed from the substituted audio, so the embedding is consistent with the audio it accompanies but not with the label attached to it.

Neither the affected speaker nor the affected index is disclosed here. Locating the record is a well-posed exercise in outlier detection: the corrupted item is detectable in embedding space, for instance by its cosine similarity to the centroid of its own labelled speaker, and more than one detection strategy will succeed. The exercise generalises to the practical problem of finding mislabelled records in a corpus that is too large to audit by listening.

Instructors who require the ground-truth index should obtain it from the dataset maintainers rather than from this document.

Known characteristics and caveats

• —Speaker and domain are confounded. The human speaker's material is conversational customer-support dialogue, whereas the synthetic speakers read expository prose. A classifier trained on this corpus may therefore separate the speakers partly on linguistic and channel grounds rather than on voice alone. Any claim about speaker modelling drawn from this data should be qualified accordingly.
• —Heterogeneous native sampling rates. The declared feature rate of 16 kHz is a decoding target applied by the datasets library. The underlying WAV files are stored at their original rates, which differ between speakers and, within one speaker, between subsets.
• —Sequential ordering. Because records are grouped by speaker, a partition taken by contiguous index yields splits that contain only one or two speakers. Shuffle, or stratify by speaker_id, before partitioning.
• —Wide duration range. Utterances span 0.72 to 54.48 seconds. Batching without length bucketing or truncation will be inefficient, and embeddings of the longest utterances are subject to the 20-second truncation described above.
• —Transcript fidelity is not certified. The text field has not been verified against the audio by forced alignment, and for the human speaker it should be treated as an approximate transcript rather than as a gold reference.
• —Balance. The three speakers are balanced by utterance count (1,500 each) but not by total duration.
• —No held-out split. Only train is provided.

Loading

from datasets import load_dataset

ds = load_dataset("aiacademy-kg/audio-3-speaker-datase-v2", split="train")

row = ds[0]
samples = row["audio"].get_all_samples()      # torchcodec API, datasets >= 4.0
waveform, sr = samples.data, samples.sample_rate
# on datasets < 4.0: waveform, sr = row["audio"]["array"], row["audio"]["sampling_rate"]

print(row["speaker_id"], row["text"][:60], len(row["wavlm_embedding"]))

Because the embeddings are unit-norm, similarity work needs no audio decoding at all:

import numpy as np

emb = np.array(ds.select_columns("wavlm_embedding")["wavlm_embedding"])
spk = np.array(ds["speaker_id"])

similarity = emb @ emb.T                       # cosine similarity, embeddings are L2-normalised
centroids = {s: emb[spk == s].mean(0) for s in np.unique(spk)}

To avoid decoding the audio when only the metadata is required, load the Parquet files directly with pyarrow and select the columns of interest.

Attribution

The embedding extractor is the work of Orange SA and is distributed under CC BY-SA 3.0. If the embeddings are used in published work, please cite:

Gengembre, N., Le Blouch, O., Gendrot, C. (2024). Disentangling prosody and timbre embeddings via voice conversion. Proc. Interspeech 2024, 2765–2769. doi:10.21437/Interspeech.2024-207

@inproceedings{gengembre24_interspeech,
  title     = {Disentangling prosody and timbre embeddings via voice conversion},
  author    = {Nicolas Gengembre and Olivier {Le Blouch} and C\'edric Gendrot},
  year      = {2024},
  booktitle = {Interspeech 2024},
  pages     = {2765--2769},
  doi       = {10.21437/Interspeech.2024-207},
  issn      = {2958-1796},
}

Licensing and intended scope

This corpus is released for teaching purposes. The synthetic portions were generated with a third-party text-to-speech system and remain subject to that system's terms of use; the recorded portion was contributed for educational use. Redistribution terms should be confirmed with the maintainers before the data is used outside a teaching context, and the corpus is not intended for the enrolment of speakers in biometric systems.