sohaibdevv/llm-evaluation-framework

1---2license: mit3language:4  - en5tags:6  - llm-evaluation7  - benchmarking8  - nlp9  - evaluation10  - accuracy11  - hallucination12  - reasoning13  - gpt14  - claude15  - gemini16  - mistral17  - llama18  - mmlu19  - truthfulqa20  - open-source21  - python22  - fastapi23  - streamlit24library_name: llm-evaluation-framework25pipeline_tag: text-generation26---27 28# LLM Evaluation Framework29 30<p align="center">31  <img src="https://img.shields.io/badge/python-3.10%2B-22c55e?style=flat-square&logo=python&logoColor=white"/>32  <img src="https://img.shields.io/badge/License-MIT-eab308?style=flat-square"/>33  <img src="https://img.shields.io/badge/FastAPI-0.115-14b8a6?style=flat-square&logo=fastapi"/>34  <img src="https://img.shields.io/badge/Streamlit-1.40-ef4444?style=flat-square&logo=streamlit"/>35  <img src="https://img.shields.io/badge/LiteLLM-1.52-8b5cf6?style=flat-square"/>36  <img src="https://img.shields.io/github/stars/sohaibdevv/LLM-Evaluation-Framework?style=flat-square&color=eab308"/>37</p>38 39> **Production-grade open-source LLM benchmarking.**40> Evaluate GPT-4, Claude, Gemini, Mistral and Llama on 5 metrics — side by side — in one command.41 42## What This Is43 44This is the **model card / hub page** for the LLM Evaluation Framework.45The framework itself is a Python tool, not a neural network weight — this page serves as46the HuggingFace hub entry point linking all resources together.47 48| Resource | Link |49|---|---|50| GitHub | https://github.com/sohaibdevv/LLM-Evaluation-Framework |51| Live Demo | https://huggingface.co/spaces/sohaibdevv/llm-eval-demo |52| Dataset | https://huggingface.co/datasets/sohaibdevv/llm-eval-benchmark |53| Docs | https://sohaibdevv.github.io/LLM-Evaluation-Framework/ |54 55## Quick Start56 57```bash58pip install llm-evaluation-framework59export OPENAI_API_KEY="sk-..."60llm-eval run --model gpt-4o-mini --benchmark mmlu --samples 10061```62 63**Output:**64```65╭──────────────────────────────────────╮66│  Evaluation: gpt-4o-mini             │67├──────────────────┬───────────────────┤68│ Accuracy         │ 78.00%            │69│ Avg Latency      │ 432 ms            │70│ P95 Latency      │ 1240 ms           │71│ Total Cost       │ $0.0023           │72│ Hallucination    │ 2.40%             │73│ Reasoning Score  │ 7.2 / 10          │74╰──────────────────┴───────────────────╯75```76 77## 5 Evaluation Metrics78 79| Metric | Description | Output |80|---|---|---|81| **Accuracy** | 4-strategy cascade: exact → normalized → MC → fuzzy | 0.0–1.0 |82| **Latency** | p50, p75, p90, p95, p99 percentiles + SLA violation rate | ms |83| **Cost** | Real token counts × pricing table for 15+ models | $/1K tokens |84| **Hallucination Rate** | Linguistic signal analysis (v1), NLI planned (v2) | 0.0–1.0 |85| **Reasoning Quality** | Chain-of-thought depth scoring | 1–10 |86 87## Supported Models88 89| Provider | Models |90|---|---|91| OpenAI | GPT-4o, GPT-4o-mini, o1, o1-mini, GPT-3.5-turbo |92| Anthropic | Claude 3.5 Sonnet, Claude 3.5 Haiku, Claude 3 Opus |93| Google | Gemini 1.5 Pro, Gemini 1.5 Flash, Gemini 2.0 Flash |94| Mistral | Mistral Large, Mistral Small |95| Meta | Llama 3 70B, Llama 3 8B (via Together AI) |96| Local | Ollama, vLLM, HuggingFace TGI |97 98## Sample Benchmark Results (MMLU, 100 samples)99 100| Model | Accuracy | Latency | Cost/1K | Hallucination | Reasoning |101|---|---|---|---|---|---|102| GPT-4o | 88.2% | 892ms | $0.0080 | 1.8% | 8.4/10 |103| Claude 3.5 Sonnet | 87.6% | 1240ms | $0.0090 | 2.1% | 8.6/10 |104| GPT-4o-mini | 78.4% | 432ms | $0.0003 | 3.2% | 7.2/10 |105| Gemini 1.5 Flash | 76.8% | 380ms | $0.0001 | 4.1% | 6.8/10 |106| Claude 3 Haiku | 74.2% | 410ms | $0.0010 | 4.8% | 6.5/10 |107 108**Key finding:** GPT-4o-mini achieves 88% of GPT-4o's accuracy at 4% of the cost.109 110## Features111 112- **Async parallel evaluation** — 10 models at once via `asyncio.Semaphore`113- **Streamlit dashboard** — radar charts, latency histograms, cost vs quality scatter114- **FastAPI REST API** — 12 endpoints with OpenAPI docs115- **CLI tool** — 7 subcommands with rich terminal output116- **PDF report generator** — professional layout via ReportLab117- **SQLite persistence** — zero-config, file-based storage118- **Docker ready** — multi-stage build, `docker-compose up`119- **40+ tests, 95% coverage** — pytest, no API keys needed120 121## Architecture122 123```124CLI / FastAPI / Streamlit / PDF Generator125              │126        Core Evaluator (asyncio)127              │128   ┌──────────┼──────────┬──────────┐129Metrics  Benchmarks  Database  LiteLLM130accuracy  MMLU        SQLite    OpenAI131latency   TruthfulQA           Anthropic132cost      Custom CSV           Google133hallucin.                      Mistral134reasoning                      Together135```136 137## Install138 139```bash140# pip141pip install llm-evaluation-framework142 143# With extras144pip install "llm-evaluation-framework[dashboard,reports,dev]"145 146# Docker147docker-compose up -d148```149 150## License151 152MIT — free for research and commercial use.153 154## Citation155 156```bibtex157@software{sohaibdevv_llm_eval_2025,158  author  = {sohaibdevv},159  title   = {LLM Evaluation Framework},160  year    = {2025},161  url     = {https://github.com/sohaibdevv/LLM-Evaluation-Framework},162  license = {MIT}163}164```165