egroupai/realworld-ai-support-dialog-benchmark-v1

Real-World AI Support Dialog Benchmark v1

This dataset is a synthetic but realistic benchmark for evaluating AI assistants in support workflows.

Why this dataset exists

Many AI demos are too toy-like to reflect production support conversations. This benchmark simulates realistic support cases with:

• —Ambiguous user intent
• —Multi-turn clarifications
• —Policy constraints (refund windows, account security, compliance)
• —Escalation and handoff conditions
• —Hallucination-risk prompts

What is included

• —ai_support_dialogs_v1.csv
• —120 synthetic support scenarios and expected assistant behaviors
• —evaluation_labels_v1.csv
• —Quality labels for each scenario (accuracy, safety, actionability, tone)
• —baseline_results_v1.csv
• —Example baseline scores from prompt variants
• —prompt_templates_v1.md
• —Prompting templates used for simulation and evaluation

Data schema

ai_support_dialogs_v1.csv

• —scenario_id: unique scenario ID
• —domain: support domain (billing, operations, onboarding, account, troubleshooting)
• —user_goal: user intent summary
• —user_message: representative user request
• —context_snippet: non-sensitive context supplied to assistant
• —expected_actions: expected action list
• —safe_response_example: example of safe and useful response
• —risk_flags: risk indicators (policyrisk, privacyrisk, hallucination_risk)

evaluation_labels_v1.csv

• —scenario_id
• —factuality_score (1-5)
• —policy_alignment_score (1-5)
• —helpfulness_score (1-5)
• —tone_score (1-5)
• —overall_pass (true/false)
• —review_notes

Design principles

1. 1.No proprietary architecture disclosure
2. 2.No internal endpoint, credential, or deployment details
3. 3.No customer-identifiable information
4. 4.Evaluation-first structure for reproducible benchmarking

Intended use

• —Benchmarking support assistant prompts
• —Regression testing for assistant quality
• —Demonstrating practical AI support use cases

Limitations

• —Scenarios are synthetic and may not cover all edge cases.
• —Baseline scores are examples and should not be interpreted as universal performance.

License and citation

This dataset is published for open sharing and evaluation use. Please cite the dataset repository URL when reusing.