mazesmazes/turn-end-detection

Turn-end detection from real ASR prefixes, with audio

100,348 labelled end-of-turn decision points over 53,140 synthesized customer-service utterances, each one paired with the 16 kHz audio it was cut from, plus the endpointing decisions 6 commercial endpointer configurations made on the same audio.

The question each row poses is the one a voice agent has to answer continuously: given everything heard so far, has the caller finished speaking? Ending the turn too early talks over the caller and can capture the wrong data; ending it too late makes the agent feel slow.

How it was made

Utterances were generated by an LLM conditioned on a task pool, spoken through text-to-speech at 16 kHz, then transcribed by running synchronous ASR on prefixes of the audio: sync(audio[0:t]) is what the recognizer makes of the audio available at instant t, which is what a streaming partial is. Verified against real streaming partials on identical audio, the text at matched timestamps is identical or near-identical, including the recognizer's own errors.

Prefixes rather than streaming partials because the decision points then belong to the dataset rather than to a decoder's flush cadence: a negative can be placed exactly at a chunk boundary and exactly before an essential datum, which is where the interesting cases are.

Every text field is therefore real ASR output, not simulated, including its mistakes -- 90007 for 90028, K-O-W-A-L-C-Z-I-K for ...Y-K.

Configs

Three tables, because several prefixes of one utterance share one audio file and inlining audio per row would carry every waveform roughly twice.

Join on `turn_key`.

from datasets import load_dataset

obs = load_dataset("mazesmazes/turn-end-detection", "observations", split="test")
turns = load_dataset("mazesmazes/turn-end-detection", "turns", split="test")

datasets 4.x decodes the Audio feature through `torchcodec`, so pip install torchcodec is needed to read turns. If you would rather not add it, take the FLAC bytes directly -- the file is lossless either way:

import io, soundfile as sf
from datasets import load_dataset, Audio

turns = load_dataset(REPO, "turns", split="test").cast_column("audio", Audio(decode=False))
samples, rate = sf.read(io.BytesIO(turns[0]["audio"]["bytes"]), dtype="float32")

The single most important field

observations.at_s is the instant the prefix was cut. A row describes turns.audio[:at_s], not the whole file. Scoring a detector on the full waveform while reading a prefix's label measures nothing.

import numpy as np
by_key = {t["turn_key"]: t for t in turns}
row = obs[0]
turn = by_key[row["turn_key"]]
audio = turn["audio"]["array"][: int(row["at_s"] * turn["audio"]["sampling_rate"])]
# `audio` is what the caller had uttered; `row["label"]` is whether they were done.

Labels

label = 1 means the caller has finished. A point is positive when its transcript covers the final chunk, measured by word recall with the essential datum excluded -- coverage rather than arrival time, because the label must describe what a detector could infer from the text in front of it. If the speaker has stopped but the recognizer is still catching up, firing still loses words, and only coverage says so.

Time then guards the positives. A recognizer will happily transcribe a plausible entity from incomplete audio: one prefix at 7.77 s carried a complete address ending 90210 while the speaker had not finished saying 90028 until 9.46 s. Coverage alone scores that 1.0 and would label the most dangerous moment in the clip a positive.

kind -- what sort of decision point it is

• —`payload_cut` -- truncated just before the essential datum. Reads finished and loses the address or order number the agent just asked for. The costliest failure, and the reason this dataset exists.
• —`trailing_frag` -- a complete request plus the start of a new thought. Punctuation offers nothing; only the dangling syntax betrays it.
• —`mid_thought_cut` -- an interior boundary whose continuation finishes the same thought.
• —`word_cut` -- a mid-phrase prefix. The easy control.
• —`chunk_cut` -- an interior boundary at a complete sentence. Labelled 1. Relabelling this from negative to positive was the single largest correctness fix in the build.
• —`complete` -- the caller has finished.

Splits are by conversation, never by example

Several prefixes come from one utterance and share nearly all their tokens, so a per-example split would score memorization and look excellent while generalizing not at all. Assignment is a hash of `domain/turn_key`, so adding data later leaves existing membership unchanged rather than silently redefining "held out".

The same correlation makes plain McNemar anti-conservative on this data (~2.5x too narrow, measured); resample turns, not rows.

endpoints: what the commercial services actually did

The same audio streamed in realtime through five configurations, recording when each declared the turn over. Realtime pacing is mandatory -- these endpointers' silence thresholds are wall-clock timers, so pushing audio faster invalidates the timings.

first_endpoint_s is the first declaration, which is the one that ends the turn in production; a later one is a turn the service opened after cutting the caller off. Using the last makes an early cut look like a patient decision seconds later. cut_early is first_endpoint_s < audio_duration_s.

Limitations -- please read before using this

• —The audio and text are synthetic. Utterances are LLM-generated and text-to-speech spoken. There is no real caller audio here, so acoustic diversity is that of a TTS voice inventory, not of a phone channel: no codec artifacts, no crosstalk, no real room noise. A detector tuned only on this will meet conditions it has not seen.
• —Entities are invented but realistically shaped. Names, emails, phone numbers, addresses and order numbers were produced by an LLM and are not drawn from any real record. They are formatted like real ones rather than using reserved ranges (example.com, 555-01xx), so a generated string may coincidentally match a real one. Any such resemblance is accidental and carries no information about a real person.
• —`mid_thought_cut` is too rare to measure. At 0.14% of points it sits below a usable floor, and scaling the build the same way would not fix the ratio. Treat per-kind results for it as indicative only.
• —English only, and one domain family (customer service across 10 domains).
• —`assemblyai_confidence` is a silence clock, not a semantic score. Six consecutive frames ramped 0.20 -> 0.97 on identical text; only the pause length changed. It is published as a measured baseline, not as a label.

Licensing

The text was produced by an LLM and the audio by a commercial text-to-speech service. Redistribution terms for synthesized speech depend on that provider's terms of service, which is why the license is marked other rather than a permissive SPDX identifier. Check the terms that apply to your use before redistributing the audio. The annotations, labels, split assignment and benchmark measurements are offered for research use.

Citation

@misc{turn_end_detection},
  title  = {Turn-end detection from real ASR prefixes},
  author = {Kroman, Alex},
  year   = {2026},
  url    = {https://huggingface.co/datasets/mazesmazes/turn-end-detection}
}