QURAT25/ai-observability-dashboard

baaki README content yahan se shuru

Signal — AI Observability Dashboard

A real-time observability layer for LLM applications. Tracks cost, latency, and response quality across every model call — automatically.

Why

As teams integrate LLMs into production, visibility often lags behind adoption: nobody's watching per-call cost, latency spikes go unnoticed, and hallucinated responses slip through silently. Signal wraps any LLM call with automatic logging, so this data is captured without manual tracking.

Features

• —Cost tracking — per-call cost calculated from token usage and model pricing
• —Latency monitoring — response time tracked for every call
• —Quality flagging — heuristic-based detection of short, repetitive, or low-confidence responses
• —Live dashboard — auto-refreshing charts for cost trend, model usage, and flagged alerts
• —Provider-agnostic — tested with Groq; works with any LLM API that returns token usage

Stack

• —Backend: Python, FastAPI, SQLite
• —Frontend: Vanilla JS, Chart.js
• —Tested with: Groq API (Llama 3.3 70B, Llama 3.1 8B)

How it works

1. 1.app/logger.py wraps any LLM call function with track_llm_call(...)
2. 2.Each call is timed, priced (app/pricing.py), and quality-checked (app/quality_check.py)
3. 3.Results are logged to SQLite (app/models.py)
4. 4.app/main.py exposes REST endpoints for aggregated stats
5. 5.dashboard.html polls those endpoints and renders live charts

Running locally

\\\bash pip install -r requirements.txt python generate_demo_data.py # optional — populates sample data uvicorn app.main:app --reload --port 8000 \\\

Then open dashboard.html in a browser, or visit http://localhost:8000 if serving it via FastAPI.

Screenshots

(yahan apna dashboard ka screenshot add karna — neeche instructions hain)

Possible extensions

• —Swap heuristic quality checks for an LLM-as-judge scoring call
• —Add Slack/email alerting on cost or latency spikes
• —Postgres backend for production persistence