CoolFace
Datasetpublic

WhissleAI/betrac-2026-with-meta

betrac-2026-with-meta Annotated speech dataset created with Whissle Annotator — a multimodal annotation pipeline for speech, NLP, and visual analysis. Source Dataset This dataset is derived from the following HuggingFace dataset(s): BeTraC/betrac-2026:0:50 BeTraC/betrac-2026:50:50 BeTraC/betrac-2026:100:50 BeTraC/betrac-2026:150:50 BeTraC/betrac-2026:200:50 BeTraC/betrac-2026:250:50 BeTraC/betrac-2026:300:50 BeTraC/betrac-2026:350:50 BeTraC/betrac-2026:400:50… See the full description on the dataset page: https://huggingface.co/datasets/WhissleAI/betrac-2026-with-meta.

sourceHugging Faceapache-2.0updated 5mo agoView on Hugging Face
0likes40downloads
Dataset Card

betrac-2026-with-meta

Annotated speech dataset created with [Whissle Annotator](https://experiments.whissle.ai) — a multimodal annotation pipeline for speech, NLP, and visual analysis.

Source Dataset

This dataset is derived from the following HuggingFace dataset(s):

The original source dataset contains raw audio conversations. This version segments those conversations into individual utterances and enriches each segment with 55 classification tokens covering audio characteristics, NLP annotations, and named entities — making it ready for meta-ASR model training.

Dataset Description

Full BeTraC dataset with Whisper forced alignment and SPEAKER_CHANGE tags

Key Statistics

MetricValue
Records57,917
Total Audio464.6 hours
Avg Segment28.9s
Entity Coverage100%
Entity Types397
Classification Tokens55 per record

Pipeline Stages Used

  • —s04_audio_classify
  • —s05_entity_intent
  • —s06_visual_extract
  • —s07_visual_classify
  • —s08_crossmodal

Experiment Schema

The annotation pipeline was configured with the following domain-specific schema (auto-generated during experiment creation):

Entity Types (80): PERSON_NAME, ORGANIZATION, LOCATION, DATE, TIME, NUMBER, ACTION_ITEM, MEETING, QUESTION, RESPONSE, FEEDBACK, COMPLAINT, ANNOUNCEMENT, REMINDER, NOTE, TASK, SCHEDULE, APPOINTMENT_DATE, APPOINTMENT_TIME, DURATION, PRIORITY, ORDER_STATUS, RETURN_STATUS, CANCELLATION_REASON, RELATIONSHIP, OCCUPATION, EDUCATION_LEVEL, EXPERIENCE, PROJECT_NAME, PRODUCT, SERVICE, BRAND, PRICE, ORDER_NUMBER, CLAIM_NUMBER, POLICY_NUMBER, EVENT, ADDRESS, CITY, STATE, COUNTRY, ZIP_CODE, ROOM_NUMBER, DEVICE_NAME, URL, IP_ADDRESS, SOFTWARE_VERSION, MODEL_NUMBER, CATEGORY, DEPARTMENT, SKILL, REVIEW, UPDATE, INVOICE_NUMBER, INVOICE_STATUS, PAYMENT_AMOUNT, PAYMENT_METHOD, PAYMENT_REFERENCE, PAYMENT_STATUS, CURRENCY, BANK_NAME, TICKET_NUMBER, BOOKING_REFERENCE, WARRANTY_PERIOD, HEALTH_STATUS, SYMPTOM, DIAGNOSIS, MEDICATION, VITAL_SIGN, MEASUREMENT, WEIGHT, HEIGHT, TEMPERATURE, DRIVERS_LICENSE, PASSPORT_NUMBER, SOCIAL_SECURITY_NUMBER, TAX_ID, LICENSE_PLATE, ACCOUNT_NUMBER, CREDIT_CARD_NUMBER

Intent Types (15): INITIATE_CONVERSATION, TERMINATE_CONVERSATION, INTERRUPT, AGREE, DISAGREE, CLARIFY, ELABORATE, SEEK_CONSENSUS, EXPRESS_EMPATHY, CHANGE_TOPIC, HOLD_TURN, YIELD_TURN, REPAIR_MISUNDERSTANDING, PROVIDE_ASSURANCE, CHALLENGE_STATEMENT

Sentiment Types (8): FRUSTRATED, CONFUSED, URGENT, HESITANT, SKEPTICAL, ENTHUSIASTIC, ANXIOUS, EMPATHETIC

Annotation Focus: Annotators should prioritize capturing the dynamics of human interaction, precisely marking speaker turns and changes, and identifying behavioral cues reflected in both verbal and non-verbal communication.

NLP Guidance: For NLP, annotators must pay close attention to the turn-taking structure, identifying explicit and implicit speaker intents beyond simple information exchange. Focus on discourse markers, hedges, and backchannels that indicate conversational engagement and agreement/disagreement. Special attention should be given to how language patterns reveal underlying behavioral states, such as hesitation, assertiveness, or emotional shifts.

Audio Guidance: Audio annotators must precisely mark SPEAKER_CHANGE events and ensure accurate Whisper forced alignment for word-level timing. Focus on identifying individual speakers, their vocal characteristics (pitch, volume, speaking rate), and paralinguistic cues like emotion shifts, disfluencies, and pauses. Background noise classification is also important to contextualize the recording environment and potential distractions.

Schema

Each record contains:

  • —`text` — Transcript with inline entity tags (e.g., ENTITY_SYMPTOM headache END)
  • —`tagged_text` — Full NeMo meta-ASR format: transcript with entities | TOKEN1 TOKEN2 ...
  • —`audio` — Audio waveform (16kHz WAV)
  • —`duration` — Segment duration in seconds
  • —Classification fields — audio_* (age, gender, emotion, speech rate, etc.), nlp_* (intent, sentiment, domain, etc.)

Entity Types Found (397)

ABSTRACT, ABSTRACT_CONCEPT, ACCOMMODATION, ACCOUNT_NUMBER, ACKNOWLEDGE, ACKNOWLEDGEMENT, ACKNOWLEDGMENT, ACTION, ACTION_ITEM, ACTIVITY, ADDRESS, ADVICE, AGE, AGREEMENT, ALLERGEN, AMOUNT, ANATOMICAL_LOCATION, ANATOMICAL_PART, ANATOMICAL_STRUCTURE, ANATOMY, ANIMAL, ANNOUNCEMENT, APOLOGIZE, APOLOGY, APPLICATION, APPOINTMENT, APPOINTMENT_DATE, APPOINTMENT_DATE_TIME, APPOINTMENT_INFORMATION, APPOINTMENT_TIME...

Classification Tokens

The tagged_text field uses a pipe separator followed by 55 classification tokens in fixed order:

transcript with ENTITY_SYMPTOM cough END | AGE_30_45 GENDER_MALE EMOTION_NEUTRAL ... DOMAIN_MEDICAL SPAM_HAM ...

Audio tokens: age, gender, emotion, intensity, speechrate, volume, pitch, disfluency, noise, speakercount, etc. NLP tokens: intent, sentiment, topic, speech_act, formality, politeness, complexity, domain, etc.

Usage

Load with HuggingFace Datasets

python
from datasets import load_dataset

ds = load_dataset("WhissleAI/betrac-2026-with-meta")
print(ds["train"][0])
# {'text': 'ENTITY_SYMPTOM headache END for three days...', 'audio': {'array': [...], 'sampling_rate': 16000}, ...}

NeMo Meta-ASR Fine-Tuning

Use with PromptingNemo for meta-ASR model training:

python
from datasets import load_dataset

ds = load_dataset("WhissleAI/betrac-2026-with-meta")

# Write NeMo manifest
with open("manifest.json", "w") as f:
    for row in ds["train"]:
        import json, soundfile as sf
        audio_path = f"audio/{row['sample_id']}.wav"
        sf.write(audio_path, row["audio"]["array"], row["audio"]["sampling_rate"])
        f.write(json.dumps({
            "audio_filepath": audio_path,
            "text": row["tagged_text"],
            "duration": row["duration"],
        }) + "\n")

Then use the manifest with NeMo ASR training scripts.

Annotation Process

This dataset was created using Whissle Annotator, which runs the following pipeline:

  1. 1.Source ingestion — Download and segment audio from HuggingFace datasets
  2. 2.Audio classification — Age, gender, emotion, speech rate, SNR via wav2vec2/HuBERT models
  3. 3.Entity & intent detection — Named entity recognition and intent classification via Gemini
  4. 4.Quality validation — Automated quality scoring and consistency checks
  5. 5.Text rewriting — Natural speech form normalization
  6. 6.Finalization — Merge all annotations into unified records

License

Apache 2.0