Metafazer/finrag-backend
0
1---2title: FinRAG3emoji: 📈4colorFrom: blue5colorTo: indigo6sdk: docker7app_port: 78608pinned: false9---10 11# FinRAG12 13> A production-grade, citation-enforced financial research assistant over SEC filings and earnings call transcripts.14 15 16[](https://github.com/MetaFazer/Finrag/actions/workflows/quality-gate.yml)17[](https://www.python.org/downloads/)18[](https://opensource.org/licenses/MIT)19 20---21 22## What This Does23 24FinRAG answers questions about SEC filings (10-K, 10-Q, 8-K) and earnings call transcripts. Every answer is grounded in a specific paragraph from a specific filing, with company, period, section, and page attached. When evidence doesn't support a claim, the system **refuses to answer** rather than hallucinate.25 26### Key Capabilities27 28- **Citation-enforced answers** — every claim maps to a source chunk with filing reference, section, and page29- **Hybrid retrieval** — BM25 sparse + dense vector search fused with Reciprocal Rank Fusion30- **Cross-encoder reranking** — precision-focused second-stage reranking31- **Multi-turn conversations** — entity tracking, reference resolution, session memory32- **Guardrails** — prompt injection detection, PII filtering, output validation33- **Streaming API** — Server-Sent Events for progressive UI rendering34- **Distributed tracing** — Langfuse integration with per-request cost tracking35- **Automated evaluation** — 50-item golden dataset, RAGAS metrics, LLM-as-Judge citation scoring36- **CI quality gates** — builds fail if faithfulness < 0.85 or citation coverage < 0.9037 38---39 40## Architecture41 42```43┌─────────────────────────────────────────────────────────────────┐44│ FastAPI Layer │45│ POST /query │ POST /query/stream │ GET /metrics │46└──────┬────────┴──────────┬───────────┴──────────┬──────────────┘47 │ │ │48 ▼ ▼ ▼49┌──────────────────────────────────────────────────────────────┐50│ LangGraph Orchestration │51│ │52│ ┌──────────┐ ┌──────────┐ ┌──────────┐ ┌──────────┐ │53│ │ Input │──▶│ Retrieve │──▶│ Rerank │──▶│ Route │ │54│ │ Guard │ │ (Hybrid)│ │ (Cross- │ │ (Keyword │ │55│ │ │ │ │ │ Encoder) │ │ Router) │ │56│ └──────────┘ └──────────┘ └──────────┘ └────┬─────┘ │57│ │ │58│ ┌─────────────────┬───────────────┘ │59│ ▼ ▼ │60│ ┌──────────┐ ┌──────────┐ │61│ │ Generate │ │ Calculate │ │62│ │ (Gemini) │ │ (Gemini) │ │63│ └────┬─────┘ └────┬─────┘ │64│ │ │ │65│ ▼ ▼ │66│ ┌──────────┐ ┌──────────┐ │67│ │ Validate │──▶│ Output │ │68│ │ Citations│ │ Guard │ │69│ └──────────┘ └──────────┘ │70└──────────────────────────────────────────────────────────────┘71 │ │72 ▼ ▼73┌──────────────┐ ┌──────────────────┐74│ ChromaDB │ │ Langfuse │75│ + BM25 │ │ Tracing │76│ Vector Store│ │ + Metrics │77└──────────────┘ └──────────────────┘78```79 80---81 82## Tech Stack83 84| Component | Technology |85|-----------|-----------|86| Orchestration | LangGraph (state machine with conditional routing) |87| Vector Store | ChromaDB (persistent, metadata-filtered) |88| Sparse Retrieval | BM25 via `rank-bm25` |89| Dense Retrieval | `sentence-transformers` (all-MiniLM-L6-v2) |90| Reranking | Cross-encoder (`ms-marco-MiniLM-L-6-v2`) |91| Generation | Google Gemini 2.0 Flash via `langchain-google-genai` |92| API | FastAPI + SSE (`sse-starlette`) |93| Guardrails | Custom regex + policy-based input/output guards |94| Observability | Langfuse (traces, spans, token costs) |95| Evaluation | RAGAS metrics + LLM-as-Judge citation scorer |96| Config | `pydantic-settings` + versioned YAML prompts |97| CI | GitHub Actions (lint → test → eval gate) |98 99---100 101## Setup102 103### Prerequisites104 105- Python 3.11+106- Google API key (for Gemini LLM)107 108### Installation109 110```bash111# Clone the repo112git clone https://github.com/MetaFazer/Finrag.git113cd finrag114 115# Create virtual environment116python -m venv .venv117 118# Activate (Windows)119.venv\Scripts\activate120# Activate (macOS/Linux)121source .venv/bin/activate122 123# Install with dev dependencies124pip install -e ".[dev]"125```126 127### Environment Configuration128 129```bash130# Copy example env file131cp .env.example .env132```133 134Edit `.env` with your credentials:135 136```env137# Required: Google Gemini API key138GOOGLE_API_KEY=your_key_here139 140# Optional: Langfuse observability141LANGFUSE_PUBLIC_KEY=pk-lf-...142LANGFUSE_SECRET_KEY=sk-lf-...143 144# Optional: API authentication145FINRAG_API_KEY=your_api_secret146```147 148---149 150## Quick Start151 152### 1. Ingest a Filing153 154```bash155# Download and process Apple's latest 10-K156python scripts/ingest.py --ticker AAPL --filing-type 10-K --count 1157```158 159This downloads the filing from SEC EDGAR, parses sections, chunks with metadata, and indexes into ChromaDB + BM25.160 161### 2. Start the API Server162 163```bash164uvicorn finrag.api.app:app --reload --port 8000165```166 167### 3. Query the Pipeline168 169```bash170# Synchronous query171curl -X POST http://localhost:8000/api/v1/query \172 -H "Content-Type: application/json" \173 -d '{"query": "What was Apple total net revenue for fiscal year 2024?"}'174 175# Streaming query (SSE)176curl -X POST http://localhost:8000/api/v1/query/stream \177 -H "Content-Type: application/json" \178 -d '{"query": "What was Apple total net revenue for fiscal year 2024?"}'179```180 181### 4. Check Metrics182 183```bash184curl http://localhost:8000/api/v1/metrics185```186 187---188 189## API Reference190 191| Endpoint | Method | Description |192|----------|--------|-------------|193| `/api/v1/query` | POST | Synchronous JSON response |194| `/api/v1/query/stream` | POST | Server-Sent Events streaming |195| `/api/v1/sessions/{id}` | GET | Session state inspection |196| `/api/v1/sessions/{id}` | DELETE | Clear a session |197| `/api/v1/config/prompts` | GET | Active prompt versions |198| `/api/v1/metrics` | GET | Production metrics (p50/p95 latency, costs, rates) |199 200### Query Request201 202```json203{204 "query": "What was Apple's free cash flow in FY2024?",205 "session_id": "optional-session-id",206 "metadata_filter": {"ticker": "AAPL"}207}208```209 210### Query Response211 212```json213{214 "answer": "Apple's free cash flow in FY2024 was...",215 "citations": [216 {217 "chunk_id": "abc123",218 "filing_reference": "AAPL 10-K FY2024, Item 7 - MD&A",219 "section": "Item 7",220 "relevance_score": 0.92221 }222 ],223 "session_id": "auto-generated-uuid",224 "confidence": 0.87,225 "route": "retrieve",226 "prompt_version": "v2",227 "metadata": {228 "request_id": "uuid",229 "trace_id": "langfuse-trace-id",230 "total_latency_ms": 1250231 }232}233```234 235---236 237## Evaluation238 239### Golden Dataset240 24150 manually verified Q/A pairs across 4 categories:242 243| Category | Count | Description |244|----------|-------|-------------|245| Numerical Extraction | 15 | Direct financial data queries |246| Multi-hop Comparison | 12 | Cross-document reasoning |247| Contradiction Detection | 11 | Narrative vs. data consistency |248| Out-of-scope | 12 | Should produce decline, not hallucination |249 250### Run Evaluations251 252```bash253# RAGAS metrics (faithfulness, relevancy, precision, coverage)254python -m finrag.evaluation.run_eval --mode ragas --threshold 0.85255 256# LLM-as-Judge citation scoring257python -m finrag.evaluation.run_eval --mode judge --threshold 0.90258 259# Full evaluation (both)260python -m finrag.evaluation.run_eval --mode full --output report.json261 262# Filter by category263python -m finrag.evaluation.run_eval --mode ragas --category numerical264```265 266### CI Quality Gates267 268Every PR triggers the [quality gate workflow](.github/workflows/quality-gate.yml):269 270```271lint → unit tests (60% coverage) → RAGAS eval (≥0.85) → Judge eval (≥0.90)272```273 274Builds fail if quality thresholds are not met.275 276---277 278## Project Structure279 280```281finrag/282├── .github/workflows/ # CI quality gate283│ └── quality-gate.yml284├── configs/ # Versioned prompt configs (YAML)285├── scripts/286│ └── ingest.py # EDGAR ingestion CLI287├── src/finrag/288│ ├── ingestion/ # EDGAR client, section chunker289│ ├── vectorstore/ # ChromaDB store290│ ├── retrieval/ # BM25, hybrid retriever291│ ├── orchestration/ # LangGraph, nodes, routing, memory292│ ├── guardrails/ # Input/output guards293│ ├── api/ # FastAPI app, routes, middleware, MCP294│ ├── observability/ # Langfuse tracer, metrics295│ └── evaluation/ # Golden dataset, RAGAS, LLM-as-Judge296├── tests/ # 16 test modules, 300+ tests297├── ROADMAP.md # 15-day build roadmap298├── DEBT_LEDGER.md # Technical debt tracking299└── pyproject.toml # Dependencies and tooling config300```301 302---303 304## Development305 306### Run Tests307 308```bash309# All tests310python -m pytest tests/ -v --tb=short311 312# Specific day/module313python -m pytest tests/test_integration.py -v314 315# With coverage316python -m pytest tests/ --cov=finrag --cov-report=term-missing317```318 319### Lint320 321```bash322ruff check src/ tests/323ruff format src/ tests/324```325 326### Environment Variables327 328| Variable | Required | Description |329|----------|----------|-------------|330| `GOOGLE_API_KEY` | Yes | Google Gemini API key |331| `LANGFUSE_PUBLIC_KEY` | No | Langfuse tracing (public key) |332| `LANGFUSE_SECRET_KEY` | No | Langfuse tracing (secret key) |333| `FINRAG_API_KEY` | No | API bearer token authentication |334| `FINRAG_INIT_PIPELINE` | No | Set `false` to skip pipeline init (testing) |335 336---337 338## Build Timeline339 340This project was built in 15 days following a structured roadmap:341 342| Phase | Days | Focus |343|-------|------|-------|344| Foundation | 1–3 | EDGAR ingestion, chunking, vector store |345| Retrieval | 4–6 | BM25, hybrid fusion, cross-encoder reranking |346| Generation & Safety | 7–10 | LangGraph, citations, guardrails, memory |347| API & Observability | 11–12 | FastAPI, SSE, Langfuse tracing |348| Evaluation & CI | 13–15 | Golden dataset, RAGAS, LLM-as-Judge, CI gates |349 350See [ROADMAP.md](ROADMAP.md) for full details and [DEBT_LEDGER.md](DEBT_LEDGER.md) for known technical debt.351 352---353 354## Deployment355 356This project uses a two-branch strategy:357 358| Branch | Purpose | Vector Store | Reranker Model |359|----------|----------------------|-------------------|-------------------------|360| `main` | Local development | ChromaDB local | MiniLM-L-6-v2 |361| `deploy` | Cloud deployment | ChromaDB Cloud | MiniLM-L-2-v2 |362 363### Live Demo364- **Frontend:** [your-app.vercel.app](https://your-app.vercel.app)365- **Backend:** Deployed on Render free tier366- **Vector store:** ChromaDB Cloud free tier367 368> **Note on cold starts:** The backend is hosted on Render's free tier.369> If the service has been inactive, the first request may take 20–30370> seconds to wake up. Subsequent requests are fast. This is a known371> free-tier constraint managed with a keepalive ping every 14 minutes.372 373### Running Locally374Checkout `main` branch and follow the setup instructions above.375Local setup uses a persistent ChromaDB instance and the full376L-6 reranker model with no cold start constraints.377 378---379 380## License381 382MIT383