Nav772/llm-evaluation-dashboard

🧪 LLM Evaluation Dashboard

Compare the performance of multiple Large Language Models across reasoning, knowledge, and instruction-following tasks using the HuggingFace Inference API.

🎯 What This Does

1. 1.Benchmark Results — View pre-computed evaluation results across 15 tasks
2. 2.Interactive Charts — Visualize accuracy and latency comparisons
3. 3.Live Testing — Test any model with your own custom prompts
4. 4.Detailed Analysis — Filter and explore results by model and category

🤖 Models Evaluated

📊 Evaluation Categories

1. Reasoning (Math & Logic)

Tests mathematical computation and logical deduction abilities.

Example tasks:

• —"A store sells apples for $2 each. If I buy 3 apples and pay with $10, how much change do I get?"
• —"If all roses are flowers, and some flowers fade quickly, can we conclude that some roses fade quickly?"

2. Knowledge (Facts)

Tests factual accuracy across science, history, and geography.

Example tasks:

• —"What is the chemical symbol for gold?"
• —"What planet is known as the Red Planet?"

3. Instruction Following

Tests ability to follow specific format constraints.

Example tasks:

• —"Return a JSON object with keys 'name' and 'age'"
• —"List exactly 3 colors, one per line"
• —"Write a sentence of exactly 5 words"

📈 Key Findings

Insights

• —Mistral-7B achieved the best overall accuracy (80%) with the fastest response time (0.39s avg)
• —Qwen2.5-Coder excelled at reasoning tasks despite being code-focused
• —Qwen2.5-72B had perfect instruction following but struggled with reasoning
• —Larger models ≠ better performance — 7B Mistral outperformed 70B+ models

🔧 Technical Implementation

Evaluation Pipeline

┌─────────────────────────────────────────────────────────────┐
│                  LLM Evaluation Pipeline                    │
├─────────────────────────────────────────────────────────────┤
│                                                             │
│  ┌─────────────┐    ┌─────────────┐    ┌─────────────┐     │
│  │  15 Tasks   │ →  │  5 Models   │ →  │  75 Total   │     │
│  │  3 Categories│    │  HF API     │    │  Evaluations│     │
│  └─────────────┘    └─────────────┘    └─────────────┘     │
│                                                             │
│                          ↓                                  │
│                                                             │
│  ┌─────────────────────────────────────────────────────┐   │
│  │              Scoring Functions                       │   │
│  │  • contains / contains_lower (substring match)      │   │
│  │  • json_valid (JSON parsing)                        │   │
│  │  • line_count / word_count (format validation)      │   │
│  │  • starts_with_lower (constraint checking)          │   │
│  └─────────────────────────────────────────────────────┘   │
│                                                             │
│                          ↓                                  │
│                                                             │
│  ┌─────────────────────────────────────────────────────┐   │
│  │              Dashboard Visualization                 │   │
│  │  • Accuracy bar charts                              │   │
│  │  • Category heatmaps                                │   │
│  │  • Latency comparisons                              │   │
│  │  • Filterable results table                         │   │
│  └─────────────────────────────────────────────────────┘   │
│                                                             │
└─────────────────────────────────────────────────────────────┘

Scoring Methods

Tech Stack

🚀 Live Model Comparison

The dashboard includes a Live Comparison feature where you can:

1. 1.Enter any custom prompt
2. 2.Select which models to compare
3. 3.See responses side-by-side with latency metrics

⚠️ Limitations

• —Rate Limiting: HF Inference API has rate limits; some models may timeout
• —Task Coverage: 15 tasks is a sample, not comprehensive benchmark
• —Single Run: Results from one evaluation run (no statistical averaging)

🎓 What This Project Demonstrates

• —LLM Evaluation Design — Creating meaningful benchmarks
• —API Integration — Working with HuggingFace Inference API
• —Data Visualization — Building interactive dashboards
• —Scoring Systems — Implementing automated evaluation metrics

👤 Author

[Nav772](https://huggingface.co/Nav772) — Built as part of an AI/ML Engineering portfolio.

📄 License

MIT License