Apppublic

Sidhartha2004/document-ai-backend

sourceHugging Faceupdated 3mo agoView on Hugging Face

0likes

App README

DocuMind AI (RAG Engine)

Lead Architect & Author: Sidhartha Vyas

Welcome to DocuMind AI, a production-grade Retrieval-Augmented Generation (RAG) backend engine. This system allows you to upload, enrich, chunk, embed, and index PDF documents, routing questions through an intelligent pre-retrieval Intent Classification layer, and generating responses using state-of-the-art LLMs (Gemini, OpenAI, Groq, and OpenRouter) with local embedding models.

🏗️ Architecture & Processing Pipeline

The engine combines standard document parsing with semantic vector indexing, metadata enrichment, and multi-stage retrieval pipelines. The diagram below shows the end-to-end flow of a document from upload to query resolution:

mermaid

graph TD
    %% Styling
    classDef process fill:#1d2736,stroke:#3b82f6,stroke-width:2px,color:#fff;
    classDef storage fill:#152214,stroke:#22c55e,stroke-width:2px,color:#fff;
    classDef external fill:#2c192e,stroke:#a855f7,stroke-width:2px,color:#fff;

    %% Elements
    A[PDF Document Upload] --> B[PDF Processor: pypdf + pdfplumber]:::process
    B --> C[Metadata Extractor: Info + Content Parser]:::process
    C --> D[Text Chunker: RecursiveCharacterTextSplitter]:::process
    D --> E[Embedder Service: SentenceTransformer]:::process
    E --> F[Vector Store: ChromaDB SQLite]:::storage
    
    G[User Question] --> H[Intent Classifier: Cosine Similarity]:::process
    H --> I{Greeting Intent?}
    I -- Yes --> J[Immediate Friendly Response]
    I -- No --> K[Retriever: Intent-Aware Multiplier]:::process
    F -.-> K
    K --> L[Re-ranker: CrossEncoder ms-marco]:::process
    L --> M[LLM Orchestrator: Gemini / OpenAI / Groq / OpenRouter]:::external
    M --> N[User Answer + Citations]
    
    N --> O[RAGAS Evaluator: Sampled Quality Metrics]:::external

1. The Ingestion Pipeline (Upload & Indexing)

—Two-Stage Text Extraction: The system employs a fail-safe PDF extractor. It first tries pypdf for speed. If the text extraction quality is low (e.g. scanned documents or complex multi-column structures), it falls back automatically to pdfplumber to preserve layout structure and extract tabular data.
—Rich Metadata Extraction: During ingestion, the processor extracts standard properties (title, author, subject, upload timestamp) and content-derived properties:
—Language: Automated detection using the langdetect library to determine the primary document language (e.g., en, de).
—Keywords: Dynamically parses the document to extract top semantic keywords (TF-IDF-like word frequency analysis with a custom English/technical stop-words filter) for auto-tagging.
—Headings & Sections: Scans document layouts (using regex patterns + structural font heuristics) to extract section titles and headings.
—Read Time & Word Count: Calculates total word count and estimates reading time in minutes based on standard adult reading speed of 200 WPM (configurable).
—Chunking: Text is split into overlapping chunks of 512 tokens with a 50-token overlap using a recursive character text splitter.
—Embedding: Local embeddings are generated in batches (using sentence-transformers/all-MiniLM-L6-v2), producing a 384-dimension normalized vector for each chunk.
—Vector Database: Chunks, vectors, and flattened metadata are written to ChromaDB with a persistent SQLite backend.

2. The Retrieval & Generation Pipeline (Q&A)

—Intent Classification: Before querying the database, user questions are analyzed by a lightweight local Intent Classifier that computes semantic similarity against predefined query templates using embeddings. Supported intents:
—greeting: Fast-path bypass. Generates an immediate conversational response without querying the vector DB (~2ms latency).
—summarization: Adjusts retrieval to fetch a broad range of chunks (3× multiplier by default) and applies Maximal Marginal Relevance (MMR) to maximize information coverage and minimize redundancy.
—comparison: Adjusts retrieval (2× multiplier) and uses hybrid keyword + semantic search for cross-section coverage.
—troubleshooting: Broad-scale hybrid search targeting error patterns and diagnostic codes.
—definition: Restricts retrieval to a low, precise cap (top_k ≤ 3) with tight semantic matching.
—factual_question / follow_up: Performs standard top-k retrieval + CrossEncoder re-ranking.
—Two-Stage Retrieval (Reranking): For standard queries, the system recalls a broad set of candidates (top_k * 4) from the database, then narrows them down using a Cross-Encoder model (cross-encoder/ms-marco-MiniLM-L-6-v2) which scores the exact query-chunk interaction.
—Diverse Selection (MMR): Optional MMR is used to prune redundant chunks, ensuring maximum information density within the LLM's context window.
—Multi-Provider LLM Orchestrator: Answer generation is delegated to LLMService which supports 4 different providers:
—Gemini: Accesses Google Gemini models (gemini-2.5-flash default) via google.generativeai.
—OpenAI: Accesses standard GPT models (gpt-3.5-turbo default).
—Groq: Accesses ultra-fast open-weights models (llama-3.3-70b-versatile default).
—OpenRouter: Accesses the latest free LLM models (e.g. meta-llama/llama-3.3-70b-instruct:free).
—Tenacity Resilience: The LLM service is wrapped with a retry mechanism (tenacity) to automatically handle rate limits, transient network issues, or API provider hiccups (up to 3 retries with exponential backoff).
—Sampled RAGAS Evaluation: 5% of queries are randomly evaluated in the background using the RAGAS framework (calculating faithfulness, answer relevance, and context metrics) and logged as a confidence score.

⚙️ Configuration & Environment Settings

The project uses Pydantic Settings for startup validation and reads values directly from a .env file.

Create a .env file in the root of the backend folder (based on .env.example):

ini

# ===========================================
# LLM CONFIGURATION
# ===========================================

# Choose your LLM provider: "gemini", "openai", "groq", or "openrouter"
LLM_PROVIDER=gemini
GEMINI_API_KEY=your_gemini_api_key_here

# Alternative API Keys
# OPENAI_API_KEY=sk-your-openai-api-key-here
# GROQ_API_KEY=gsk_your_groq_key_here
# OPENROUTER_API_KEY=sk-or-v1-your-openrouter-key-here

# Provider-Specific Models (Optional, defaults are free/optimized models)
# GROQ_MODEL=llama-3.3-70b-versatile
# OPENROUTER_MODEL=meta-llama/llama-3.3-70b-instruct:free
# OPENROUTER_MODEL=deepseek/deepseek-v4-flash:free                      # Powerhouse for large document search (free, 1M context)
# OPENROUTER_MODEL=xiaomi/mimo-v2.5                                     # Omnimodal perception for visual docs/media (1M context)
# OPENROUTER_MODEL=nvidia/nemotron-3-ultra-550b-a55b:free               # Massive reasoning for complex data synthesis (free, 1M context)

# Local Embedding Model (downloads automatically on first run ~80MB)
EMBEDDING_MODEL=sentence-transformers/all-MiniLM-L6-v2

# ===========================================
# DATABASE PATHS
# ===========================================
CHROMA_PERSIST_DIRECTORY=./data/chroma_db
UPLOAD_DIR=./data/uploads

# ===========================================
# CHUNKING STRATEGY
# ===========================================
CHUNK_SIZE=512
CHUNK_OVERLAP=50
MAX_CHUNKS_PER_DOC=1000

# ===========================================
# RETRIEVAL CONFIGURATION
# ===========================================
TOP_K=20
RERANK_TOP_K=5
SIMILARITY_THRESHOLD=0.3

# ===========================================
# INTENT CLASSIFICATION CONFIGURATION
# ===========================================
ENABLE_INTENT_CLASSIFICATION=True
INTENT_CONFIDENCE_THRESHOLD=0.25

# Intent-aware multipliers
SUMMARIZATION_TOP_K_MULTIPLIER=3
COMPARISON_TOP_K_MULTIPLIER=2
TROUBLESHOOTING_TOP_K_MULTIPLIER=2
DEFINITION_TOP_K_CAP=3

# ===========================================
# METADATA EXTRACTION SETTINGS
# ===========================================
READING_SPEED_WPM=200
MAX_HEADINGS_PER_DOC=50
MAX_AUTO_KEYWORDS=15

# ===========================================
# APPLICATION SETTINGS
# ===========================================
MAX_FILE_SIZE=10485760
DEBUG=True

🚀 Installation & Running

Prerequisites

—Python 3.9 - 3.12 (Python 3.12 recommended)
—pip package manager

Setup Instructions

1.Navigate to the backend project folder:

bash

   cd ai-docs-assistant

1.Install the dependencies:

bash

   pip install -r requirements.txt

1.Start the Development Server:

bash

   python run_dev.py

FastAPI will launch with uvicorn auto-reload enabled:

—Interactive OpenAPI/Swagger Docs: http://localhost:8000/docs
—Alternative ReDoc UI: http://localhost:8000/redoc
—API Health Check: http://localhost:8000/health

📡 Key API Endpoints Reference

1. Root Endpoint

—URL: /
—Method: GET
—Response Body:

json

  {
    "status": "online",
    "version": "2.0.0",
    "embedding_model": "sentence-transformers/all-MiniLM-L6-v2",
    "llm_provider": "gemini",
    "features": {
      "intent_classification": true,
      "rich_metadata_extraction": true
    }
  }

2. Health Check

—URL: /health
—Method: GET
—Response Body:

json

  {
    "status": "healthy",
    "components": {
      "vector_store": "operational",
      "llm_service": "operational",
      "embedding_service": "operational",
      "intent_classifier": "enabled"
    },
    "stats": {
      "total_chunks": 142,
      "unique_documents": 3,
      "collection_name": "documentation_chunks",
      "embedding_dimension": 384
    }
  }

3. System Statistics

—URL: /stats
—Method: GET
—Response Body:

json

  {
    "vector_store": {
      "total_chunks": 142,
      "unique_documents": 3,
      "collection_name": "documentation_chunks",
      "embedding_dimension": 384
    },
    "documents": {
      "total": 3,
      "indexed": 3,
      "processing": 0,
      "failed": 0
    },
    "config": {
      "chunk_size": 512,
      "top_k": 20,
      "llm_provider": "gemini",
      "embedding_model": "sentence-transformers/all-MiniLM-L6-v2",
      "intent_classification": true,
      "intent_confidence_threshold": 0.25
    }
  }

4. Upload Document

—URL: /upload
—Method: POST
—Content-Type: multipart/form-data
—Response Body:

json

  {
    "document_id": "doc_8f12a3bc9f...",
    "filename": "api_manual.pdf",
    "status": "processing",
    "message": "Document uploaded. Processing and metadata extraction in progress.",
    "metadata": {
      "filename": "api_manual.pdf",
      "upload_timestamp": "2026-06-23T14:28:30Z",
      "file_size": 240590,
      "num_pages": 12,
      "num_chunks": 0,
      "status": "processing",
      "document_type": null,
      "author": "Sidhartha Vyas",
      "title": "API Specification v1.0",
      "language": "en",
      "estimated_reading_time_minutes": 4.5,
      "keywords": ["auth", "token", "headers"]
    }
  }

5. List Documents

—URL: /documents
—Method: GET
—Response Body:

json

  {
    "documents": [
      {
        "document_id": "doc_8f12a3bc9f...",
        "filename": "api_manual.pdf",
        "status": "indexed",
        "num_pages": 12,
        "num_chunks": 42,
        "upload_timestamp": "2026-06-23T14:28:30.405Z",
        "title": "API Specification v1.0",
        "author": "Sidhartha Vyas",
        "subject": "Core REST Endpoints",
        "language": "en",
        "keywords": ["auth", "token", "headers", "refresh"],
        "headings": ["1. Introduction", "2. Authentication", "2.1 Grant Types"],
        "section_titles": ["Authentication Flow", "Error Codes"],
        "total_word_count": 2450,
        "estimated_reading_time_minutes": 12.25,
        "tags": [],
        "document_type": null,
        "version": null
      }
    ],
    "total": 1
  }

6. Delete Document

—URL: /documents/{document_id}
—Method: DELETE
—Response Body:

json

  {
    "message": "Document deleted",
    "chunks_removed": 42
  }

7. Query Documents (RAG)

—URL: /query
—Method: POST
—Request Body:

json

  {
    "question": "How do I refresh the access token?",
    "document_ids": ["doc_8f12a3bc9f..."],
    "language": "en",
    "top_k": 5
  }

—Response Body:

json

  {
    "question": "How do I refresh the access token?",
    "answer": "To refresh your token, send a POST request to `/oauth/token` [Source 1]...",
    "sources": [
      {
        "content": "To refresh the access token...",
        "score": 0.884,
        "metadata": {
          "document_id": "doc_8f12a3bc9f...",
          "page_number": 3,
          "section_title": "Authentication"
        }
      }
    ],
    "retrieval_time": 0.082,
    "generation_time": 1.45,
    "total_time": 1.532,
    "intent": "factual_question",
    "intent_confidence": 0.941,
    "confidence_score": 0.885
  }

8. Query Preprocess (Debug Intent)

—URL: /query/preprocess
—Method: POST
—Request Body:

json

  {
    "question": "Can you give me a summary of Chapter 2?"
  }

—Response Body:

json

  {
    "question": "Can you give me a summary of Chapter 2?",
    "intent": "summarization",
    "confidence": 0.912,
    "all_scores": {
      "greeting": 0.12,
      "summarization": 0.912,
      "comparison": 0.31,
      "troubleshooting": 0.08,
      "definition": 0.15,
      "factual_question": 0.45
    },
    "retrieval_config": {
      "strategy": "diversity",
      "top_k_multiplier": 3,
      "use_reranking": false
    }
  }

9. Evaluate Query (RAGAS)

—URL: /evaluate
—Method: POST
—URL Query Params:
—question: "What is the token expiration duration?"
—answer: "Access tokens expire in 1 hour."
—context_chunks: ["Access tokens expire in 3600 seconds..."]
—ground_truth (optional): "The token expires in 1 hour."
—Response Body:

json

  {
    "metrics": {
      "faithfulness": 1.0,
      "answer_relevance": 0.95,
      "context_precision": 0.92,
      "context_recall": 1.0,
      "ragas_score": 0.967
    },
    "report": "### RAGAS Evaluation Report\n- **Faithfulness**: 100%\n..."
  }

🛠️ Verification & Testing

To run automated checks against the application, you can execute standard testing scripts. The tests verify extraction pipelines, embedding vector dimensions, and LLM provider endpoints.

bash

# Run pytest tests
pytest tests/

Created by Sidhartha Vyas — Lead Architect, DocuMind AI