Radheshyam2006/conversational-ai-evaluator

Conversational AI Interaction Evaluator

A structured methodology and web tool for assessing quality, consistency, and reliability of AI chatbot responses — grounded in behavior verification over output verification.

Built to evaluate three systems: a real LLM (Groq/LLaMA 3.1), a rule-based Rasa simulation, and a stateless Dialogflow simulation — across a shared failure pattern taxonomy that makes findings reusable across any conversational AI system.

🚀 Live Demo & Explainer Video

• —Live Application: **Click here to try the Evaluator!**
• —Demo: **Watch the Video Explanation**

Note: Free tier deployments may take 30-50 seconds to spin up on the first request.

🛠️ Why This Stack? (Technical Decisions)

I intentionally built this tool using a lightweight, dependency-free approach:

• —Backend (Python / Flask): Keeps the evaluator engine extensible. It's effortless to integrate more LLM APIs natively using official Python SDKs compared to Next.js node wrappers.
• —Frontend (Vanilla HTML/CSS/JS): Chosen for zero-build-step simplicity and maximum load speed. No React/Node overhead needed for a tool focused purely on API interaction and grading.

Core Methodology

Traditional software testing checks exact output: did the system return the right string? This doesn't work for AI — responses are non-deterministic. The same input can produce different outputs on every run.

The solution: evaluate whether the interaction goal was achieved, not whether the output matched.

Instead of asking "is this the right answer?", we ask:

• —Was the response correct in the context of what was asked?
• —Was it complete — did it address everything the user needed?
• —Was it well-structured — organized, right length, not overwhelming?
• —Was the error transparent — when it failed, did it say so clearly?

This four-dimension model produces scores that are stable, comparable across bots, and directly actionable.

Evaluation Flow

flowchart TD
    A[Define Scenario Goal] --> B[Send Input to Bot]
    B --> C[Receive Response]
    C --> D{Goal Achieved?}
    D -- Yes --> E[Score 4 Dimensions]
    D -- No --> F[Identify Failure Dimension]
    F --> G[Match Failure Pattern]
    G --> H[Classify Severity]
    H --> E
    E --> I[Create Gap Entry]
    I --> J[Append to Gap Report]

The key design decision: severity rubric and pattern taxonomy are defined before any testing begins. This removes post-hoc bias and makes the framework repeatable by anyone.

System Architecture

graph TD
    subgraph Frontend
        UI[Web UI - Chat + Eval Panel]
    end

    subgraph Backend - Flask
        API[REST API]
        ENG[Evaluator Engine]
        RPT[Gap Report Generator]
    end

    subgraph Bot Layer
        G[Groq - LLaMA 3.1<br/>Real LLM API]
        R[Rasa Simulation<br/>Rule-based]
        D[Dialogflow Simulation<br/>Intent-based]
    end

    UI -->|POST /api/chat| API
    UI -->|POST /api/evaluate| ENG
    UI -->|GET /api/report| RPT
    API --> G
    API --> R
    API --> D
    ENG --> RPT

Evaluation Dimensions

Each interaction is scored 1–5 on four independent dimensions:

These four dimensions map directly to the four failure modes most commonly observed across all three bot systems.

Failure Pattern Taxonomy

The 6 core failure patterns documented in this framework. These are system-level patterns — they appear across multiple bot implementations, which means they reflect interaction design failures, not just implementation bugs.

flowchart LR
    subgraph Critical
        CC[Context Collapse<br/>Bot loses memory mid-session]
        SF[Silent Failure<br/>Error not surfaced to user]
    end

    subgraph Major
        HA[Hallucination under Ambiguity<br/>Confident wrong answer]
        TW[Timeout without Warning<br/>3-5s delay, no feedback]
    end

    subgraph Minor
        IR[Input Rejection<br/>Fails on unusual phrasing]
        OR[Overloaded Response<br/>Too much, too fast]
    end

Severity Criteria

Reusability Across Bot Systems

This is the core design principle that separates an evaluation framework from a one-off audit.

graph LR
    subgraph Pattern is System-Agnostic
        P[Failure Pattern:<br/>Input Rejection]
    end

    P --> G2[Found in Groq<br/>On ambiguous phrasing]
    P --> R2[Found in Rasa<br/>On non-keyword inputs]
    P --> D2[Found in Dialogflow<br/>On multi-word queries]

    G2 --> U[Universal Gap Entry<br/>Recommendation applies to all 3]
    R2 --> U
    D2 --> U

A failure pattern that appears across multiple bot systems is a universal gap — it indicates a design-level issue that no single bot implementation can solve on its own.

The gap report automatically classifies failures as universal (cross-bot) vs bot-specific, making the cross-system analysis visible in the exported JSON.

The Three Bots

Groq — LLaMA 3.1 (Real LLM)

• —Uses Groq's free API tier — genuinely non-deterministic
• —Context-aware — maintains conversation history
• —Can hallucinate under ambiguous queries
• —Best for testing: correctness failures, hallucination under ambiguity

Rasa (Simulated Rule-based)

• —Strict keyword intent matching — no NLP
• —Deliberately fails on unusual phrasing (demonstrates input rejection)
• —Context collapse after every 3 turns — silently resets
• —Overloaded response on help intent — no progressive disclosure
• —Best for testing: input rejection, context collapse, overloaded response

Dialogflow (Simulated Intent-based)

• —Stateless — every turn starts from zero, no history used
• —Hallucinates personal data (test dates, scores) with high confidence
• —Random timeout (2–4s delay) on complex multi-word queries, no warning
• —Best for testing: hallucination under ambiguity, timeout, completeness

Gap Report Format

The exported JSON follows a structured schema designed to mirror compliance audit documentation.

{
  "report_metadata": {
    "framework": "Behavior Verification over Output Verification",
    "methodology": "..."
  },
  "summary": {
    "total_interactions_evaluated": 15,
    "severity_breakdown": { "critical": 2, "major": 5, "minor": 8 },
    "average_scores": {
      "correctness": 3.2,
      "completeness": 2.8,
      "structure": 3.5,
      "error_transparency": 2.1
    },
    "failure_pattern_frequency": {
      "Input Rejection without Explanation": 6,
      "Hallucination under Ambiguity": 4,
      "Context Collapse": 3
    }
  },
  "cross_bot_analysis": {
    "universal_failure_patterns": ["Input Rejection without Explanation"],
    "bot_specific_failures": { "dialogflow": ["Hallucination under Ambiguity"] }
  },
  "gaps": [
    {
      "gap_id": "G-001",
      "bot": "rasa",
      "input": "Wanna reschedule my thang",
      "failure_patterns": ["Input Rejection without Explanation"],
      "severity": "major",
      "recommendations": ["Expand intent vocabulary with synonym sets and fuzzy matching..."]
    }
  ]
}

Setup

Requirements: Python 3.10+, a free Groq API key from console.groq.com

# 1. Clone and enter directory
cd conversational-ai-evaluator

# 2. Add Groq API key
cp .env.example backend/.env
# Edit backend/.env and add: GROQ_API_KEY=gsk_your_key_here

# 3. Run setup (installs deps + starts server)
bash setup.sh

# OR manually:
cd backend
pip install -r requirements.txt
python app.py

4. Open http://localhost:5000 in your browser.

The Rasa and Dialogflow bots work without any API key — they are simulated. Only Groq requires a key.

Recommended Test Flow

1. 1.Select a bot from the top bar
2. 2.Use a pre-built scenario from scenarios/test_scenarios.json or type your own
3. 3.After each bot response, fill in the evaluation panel on the right
4. 4.Rate all four dimensions, flag observed failure patterns, set severity
5. 5.Repeat across all three bots with the same scenario — compare results
6. 6.Click Export Gap Report to download the structured JSON

The scenario chips in the welcome screen are pre-designed to trigger specific failure patterns — use them to quickly populate the framework with meaningful evaluations.

Technologies

Framework design principle: evaluate behavior, not output. A chatbot that fails gracefully and communicates its limitations honestly is more reliable than one that produces fluent-sounding but incorrect responses.