laion/voice-pairs-genuineness-vocalburst-blend-human-eval

Voice pairs for human evaluation of the genuineness and vocal-burst-blend predictors

This dataset exists to be annotated by humans. It is not a training set. Its only purpose is to let human listeners adjudicate two automatic predictors by presenting them with the comparisons those predictors claim to be able to make:

Each sample is one pair of clips. The annotator is asked which member sounds more genuine (sets A and B) or which member's vocal burst blends more naturally (sets C and D). The predictor has already made that call; the point of the exercise is to find out how often a human agrees, and whether agreement survives when the predicted difference gets small.

2,000 pairs / 4,000 distinct audio clips, in eight WebDataset tars.

Anhören / Listen

Eine Hörprobe mit 80 dieser Paare — 20 aus jeder Teilmenge, mit Audioplayern in der Seite — steht als Space bereit: **laion/voice-pairs-listening-demo**. Innerhalb jeder Teilmenge die 10 Paare mit dem größten und die 10 mit dem kleinsten Abstand; höchstens ein Paar je Stimme, echte und synthetische Hälfte je zur Hälfte.

The scale question, and how it was resolved

The thresholds for this dataset were specified as "a difference of at least 2" for the easy pairs and "at least 1, at most 1.5" for the hard ones. Those numbers are not on the 0-1 scale of the source corpus's genuineness / blend columns, and the resolution matters enough to state in full, because two plausible readings were wrong.

The predictors do not share a scale. Measured from three independent artefacts:

1. 1.The scorer that produced the corpus clamps them differently and names the keys accordingly: genu = ...clamp(0,6) emitted as genuineness_0_6, blnd = ...clamp(0,10) emitted as blend_0_10.
2. 2.The model cards agree: genuineness "predicts a 0-6 genuineness score" (val MAE 1.00), blend "a 0-10 score for how naturally that vocal burst blends" (val MAE 2.057).
3. 3.The upstream project's own caption template writes them as genuineness <x>/6 and vocal-burst blend <x>/10.

So a single "0-10 scale" reading is right for blend and wrong for genuineness.

The corpus columns are quantile ranks, not rescaled scores. The 0-1 genuineness / blend columns in the source corpus are a per-voice quantile rank (synthetic half) and a per-(dataset x language) ECDF (real half). Measured on one source file (2,393 rows, one voice): native genuineness ran 0.00-3.20 with median 1.25 while the normalised column ran 0.09-1.00 with median 0.83. There is therefore no linear map between a corpus-scale difference and a score difference, and because the rank is computed within a speaker, every voice spans 0-1 regardless of how genuine it actually sounds. Selecting on that column would have produced a dataset whose pairs differ by a within-speaker percentile while the documentation claimed a score difference.

Resolution: the thresholds are applied literally, on each predictor's own native scale, using the raw genuineness_0_6 and blend_0_10 values recovered from the upstream source parquets and joined back to every corpus row by uid (3,147,802 rows, 0 unjoined). No rescaling of the specified numbers was necessary or applied:

A caveat the annotation should be read against. The hard band (1.0-1.5) sits at or below each predictor's own held-out error: genuineness val MAE is 1.00 on 0-6, blend val MAE is 2.057 on 0-10. Sets B and D are therefore asking humans to adjudicate differences the predictors themselves cannot reliably resolve. That is a legitimate and interesting question - it is roughly "where does this model's ordering stop meaning anything" - but a low human-agreement rate on B and D is the expected result, not necessarily evidence of a broken model.

Feasibility was counted before anything was selected. Under the literal native reading the maximum number of disjoint pairs available per cell ranged from 120,765 to 461,165 against a requirement of 250 per cell, so the reading was not chosen for convenience - every candidate cell was over-supplied by two to three orders of magnitude.

Selection rule

Within each pair, everything that could be held constant was held constant, so that the predicted score is as close to the only difference as the corpus allows:

• —same language (en or de)
• —same half - synthetic voice profiles are never paired against real recordings. This matters: the genuineness predictor separates the two halves very strongly (median 1.32 on the synthetic half against 4.66 on the real half, on 0-6), so a cross-half pair would measure "synthetic vs real", not genuineness.
• —same speaker, wherever a speaker identity exists (1,976 of 2,000 pairs). The remaining 24 pairs come from a source that ships no per-speaker key at all; those are matched on predicted age band and gender band instead and carry "speaker_matched": false in their metadata, so the weaker control stays visible.
• —same primary burst label for the blend sets - comparing the blend of a laugh against the blend of a sob is not a blend comparison.
• —durations within 4 s of each other, and every clip between 1.5 s and 20 s.
• —Both members of every C/D pair actually contain a vocal burst (burst_labels non-empty). A blend score on a clip with no burst is not measuring what the annotator is asked about.

Pairs are drawn round-robin across (language x gender band x age band) strata, capped at 6 pairs per voice/speaker key, so no handful of speakers can dominate. 1,169 distinct voice/speaker keys are represented. No clip is reused anywhere in the release - 4,000 distinct clips across 2,000 pairs, i.e. exactly two per pair.

A/B order is randomised

The higher-scoring member is A in 1,012 of 2,000 pairs (50.6%). Which member scored higher is not recoverable from the filenames or the ordering - it is recorded only in the metadata field higher_is. Annotators should be shown <pair_id>.a.flac and <pair_id>.b.flac and nothing else from the JSON.

Contents

Each pair contributes three members to its tar:

<pair_id>.a.flac     one clip of the pair
<pair_id>.b.flac     the other
<pair_id>.json       metadata (below)

Audio is FLAC, mono, decoded from the corpus's MOSS neural-codec tokens. It is lossless relative to that decode - no lossy codec was layered on top, because the annotator is being asked to judge naturalness and an MP3 artefact would sit inside the very thing under test.

Metadata fields

A verdict joins back to the prediction on pair_id plus the A/B side.

Composition

Half (synthetic voice profiles vs real recordings)

Language

Gender band — synthetic half (declared on the voice card)

Gender band — real half (VoiceNet predicted perceived gender)

Age band — synthetic half (declared on the voice card)

Age band — real half (VoiceNet predicted perceived age; no declared age exists)

Source dataset

Predicted difference (delta), as actually realised

Where age and gender come from - and where they do not

This is not uniform across the two halves, and it should not be treated as if it were.

• —Synthetic half. Age and gender are declared on the voice-profile card (a free-text age band such as "Late 40s to 50s", parsed here to a decade band). Available for 499 of the 500 voice profiles in the corpus.
• —Real half. No declared age or gender exists for these recordings at all. The bands given here are the VoiceNet predicted perceptual dimensions AGEV (Voice Age) and GEND (Perceived Gender) - i.e. how old and how masculine/feminine the voice sounds to another model, not the speaker's demographics. They are used to spread the sample and to match pairs; they are not a demographic claim and must not be reported as one.

What was excluded, and why

Publication was filtered by an allowlist of verified uid patterns, never a blocklist: an unrecognised uid counts as not publishable.

Of 3,147,802 corpus rows, 929,825 were excluded by that rule before any selection:

No mediathek or evasnippets recordings or transcripts are present. Voice profiles derived from mediathek are synthetic renderings — the audio is generated by the TTS model conditioned on a voice profile, not a mediathek recording — and are in scope; 124 of the 500 voice profiles are of that origin and 200 of the 4,000 published clips come from them.

Checked directly over every published clip, not merely over the candidate pool:

No transcripts are shipped at all. The annotation task is auditory, the specified metadata does not require text, and omitting it removes any possibility of republishing a restricted transcript.

Loading

import webdataset as wds
ds = wds.WebDataset("genuineness-real-easy-000.tar").decode()
for s in ds:
    a, b, meta = s["a.flac"], s["b.flac"], s["json"]

Licence and provenance

CC-BY-4.0. Audio is decoded from the MOSS-codec representation held in the LAION voice corpus; the real-speech half derives from emolia, kartoffelphon and MLS, the synthetic half from LAION voice profiles. Scores are predictions of the two models linked at the top of this card, not human labels - supplying the human labels is what this dataset is for.