anyreach-ai/semantic-turn-taking-benchmark

Semantic Turn-Taking Benchmark

A curated evaluation benchmark for semantic turn-taking models in voice AI. Given a conversation context, predict what action the AI agent should take: speak, listen, continue speaking, or continue listening.

Unlike acoustic-based approaches (VAD, silence detection), this benchmark tests whether a model can make turn-taking decisions from text/semantic content alone.

Action Classes

`start_speaking` vs `continue_listening` — Was the user done talking?

User: I need help with my bill
Agent: Sure I can help with that what seems to be the issue
User: I was charged twice for the same order        
→ start_speaking (user done)

User: I need help with my bill
Agent: Sure I can help with that what seems to be the issue
User: I was charged twice for                   
→ continue_listening (user not done)

`start_listening` vs `continue_speaking` — User spoke while agent was talking. Interruption or backchannel?

User: What is your refund policy
Agent: Our refund policy states that all items purchased within the last thirty days are eligible for a full refund provided that the item is in its original packaging and
User: Okay I get                       
→ start_listening (real interruption, agent should stop)

User: What is your refund policy
Agent: Our refund policy states that all items purchased within the last thirty days are eligible for a full refund provided that the item is in its original packaging and
User: uh huh                        
→ continue_speaking (backchannel, agent keeps talking)

Dataset Statistics

Subsets

ten — TEN Turn-End Detection (428 examples)

Binary subset (2 classes: start_speaking, continue_listening). Single user utterances without conversation context — tests pure end-of-utterance detection.

Source: TEN-framework/ten-turn-detection

Processing: Dropped wait class (100 examples) because start_listening requires conversation context that TEN does not provide.

swda — Switchboard Dialog Act Corpus (3,523 examples)

3-class subset (start_speaking, continue_listening, continue_speaking). Real telephone conversations with up to 5 turns of context.

Source: cgpotts/swda

Processing applied:

• —Cleaned Switchboard transcription markup ({F uh} → uh, speech repairs, etc.)
• —Dropped ambiguous tags: aa/ny/ba (agreement vs backchannel), x (non-verbal), + (syntactic continuation, 56% actually complete), ^2 (collaborative completion)
• —Kept % (abandoned) only when same speaker continues (genuinely incomplete)
• —Mapped: backchannel tags (b, bk, bh, b^m) → continue_speaking; incomplete tags → continue_listening; complete tags → start_speaking

synthetic — Curated Test Set (60 examples)

Full 4-class subset. Hand-crafted customer service conversations covering all 4 action classes at 4 difficulty levels (easy, medium, hard, ultra_hard).

Source: Created by authors

Format

Each example contains:

Benchmark Results

Baseline results using anyreach-ai/semantic-turn-taking (Qwen2.5-0.5B fine-tuned for 4-class turn-taking).

Binary (EOU vs Not-EOU)

Only start_speaking and continue_listening examples are used. Predictions mapped: start_speaking/continue_speaking → EOU, continue_listening/start_listening → Not-EOU.

Multi-class

Usage

from datasets import load_dataset

# Load all subsets
ds = load_dataset("anyreach-ai/semantic-turn-taking-benchmark")

# Load a specific subset
ten = load_dataset("anyreach-ai/semantic-turn-taking-benchmark", split="ten")
swda = load_dataset("anyreach-ai/semantic-turn-taking-benchmark", split="swda")
synthetic = load_dataset("anyreach-ai/semantic-turn-taking-benchmark", split="synthetic")

# Iterate
for example in swda:
    print(example["conversation"])
    print(f"Action: {example['action']}")

Citation

@misc{semantic-turn-taking-2026,
  title={Semantic Turn-Taking Model},
  author={Shangeth Rajaa},
  year={2026},
  publisher={Hugging Face},
  url={https://huggingface.co/anyreach-ai/semantic-turn-taking}
}

Authors

• —**Shangeth Rajaa**

License

Apache 2.0