sbdh11/rag

RAG Evaluation Platform

A tool for measuring how good a RAG pipeline is, not a chatbot. Upload some documents, index them a few different ways, search them, chat over them, then run experiments and see which setup actually retrieves and answers best.

Previews

[image] System Status Dashboard

[image] Retrieval Traces

[image] Document Management & Indexing

What you can do

• —Documents : upload PDF / Markdown / text / a web page; it gets parsed in the background.
• —Index : embed a doc with different models (all-MiniLM-L6-v2, bge-small-en-v1.5, e5-small-v2, or the Gemini / OpenAI API ones) and chunking settings. Each index is its own Qdrant collection.
• —Retrieve : run a query and watch the pipeline: vector search + BM25 → fused → cross-encoder rerank, with the scores and timing for every stage.
• —Chat : ask questions over your docs; you get a cited answer plus the chunks that produced it. (Set a GEMINI_API_KEY for real answers; without one you just get the retrieved chunks.)
• —Experiments : sweep over a grid of embedding models × chunk sizes × rerank on/off × hybrid on/off. Each cell is scored against a gold set.
• —Evaluation : a dashboard with Recall@K, MRR, nDCG, latency, and (with an LLM key) groundedness and hallucination rate, so you can compare runs.
• —Traces : every retrieval is recorded in full: chunks, scores, the prompt, tokens, latency.

Run it locally

You need Docker. make is optional.

cp .env.example .env     # optional: put GEMINI_API_KEY in it for chat + the LLM judge
make up                  # or: docker compose -f docker-compose.yml -f docker-compose.dev.yml up --build -d
make seed                # loads the sample docs, a small gold set, and a baseline sweep

Then: app at http://localhost:3000, API docs at http://localhost:8000/docs, MLflow at :5000, Qdrant dashboard at :6333/dashboard. make down to stop, make test to run the backend tests.

The first build downloads PyTorch and the ML libs (slow once); the first request that needs a model downloads it (slow once).

How it's built

Python/FastAPI backend (SQLModel + Postgres, arq + Redis for background jobs), SentenceTransformers + Qdrant + BM25 + a cross-encoder reranker, Gemini for generation by default (Ollama / Anthropic / OpenAI also work), evaluation with hand-computed retrieval metrics + an LLM judge, optional MLflow tracking. Frontend is Next.js + Tailwind + shadcn/ui + Tremor. Everything's in docker-compose.yml.

Deploy

Three options — Hugging Face Spaces (one container, free), Render + Vercel, or a single VM with docker-compose. See [DEPLOY.md](DEPLOY.md).