dev-models/MultiModel-Rag

Check out the configuration reference at https://huggingface.co/docs/hub/spaces-config-reference

🤖 Multimodal RAG Assistant (Docling-Powered)

    

A state-of-the-art Multimodal Retrieval-Augmented Generation (RAG) system built for the modern document era. This assistant doesn't just read text—it understands tables, charts, diagrams, and complex layouts using IBM's Docling and Visual Language Models.

🚀 The WOW Factor

• —🧠 Deep Document Intelligence: Powered by Docling, the system extracts semantic structures (headers, tables, lists) with extreme precision.
• —👁️ Visual Understanding: Every image in your PDF is "seen" by a VLM (Llama-3-Vision) to generate rich textual descriptions for vector indexing.
• —🔍 Hybrid Search Engine: A high-performance retrieval pipeline combining CLIP (Dense) and BM25 (Sparse) to ensure zero-miss retrieval.
• —🖼️ Visual RAG Capabilities: Directly query for charts or diagrams. The assistant "shows" you the relevant visuals alongside textual answers.
• —💡 Intelligent Query Guidance: Automatically analyzes document structure to suggest the most relevant questions for the user.
• —⚡ Blazing Fast Generation: Uses Groq's Llama-3.3-70B for near-instant, high-quality responses with full streaming support.

🛠️ Architecture Overview

The system is built on a modular, production-ready foundation:

rag-app/
├── 🌐 app.py              # Streamlit Premium Interface
├── ⚙️ config.py           # Centralized configuration
├── 📦 backend/            # Domain-driven modules
│   ├── 🛠️ parser.py       # Docling Engine + VLM Describer
│   ├── 🧠 rag.py          # Hybrid Search + RAG Orchestrator
│   ├── 💾 database.py     # MongoDB Atlas Vector Store integration
│   └── 🤖 models.py       # CLIP, LLM, and VLM Connectors
├── 📁 rag_data/           # Parsed JSON persistence
├── 🐳 Dockerfile          # Multi-stage optimized build
└── 📋 requirements.txt    # Optimized dependency tree

🏗️ Core Technology Stack

🚦 Getting Started

1. Prerequisites

• —Python 3.10+
• —A MongoDB Atlas Account (for Vector Search)
• —A Groq API Key

2. Configure Environment

Create a .env file in the root directory:

# MongoDB Credentials
MONGO_USER=your_username
MONGO_PASSWORD=your_password
MONGO_HOST=your_cluster_url.mongodb.net
MONGO_DB=rag_assistant

# API Keys
GROQ_API_KEY=gsk_your_key_here

# Optional: Full URI (overrides components above)
# MONGO_URI=mongodb+srv://...

3. Quick Run (Docker)

docker compose up --build

4. Local Setup

# Install dependencies
pip install -r requirements.txt

# Launch app
streamlit run app.py

📈 Search Optimization

• —Dense Retrieval (CLIP): Captures semantic meaning and visual similarity.
• —Sparse Retrieval (BM25): Ensures keyword matches (names, technical terms) are never missed.
• —Hybrid Weighting: Fine-tuned alpha parameter balances the two search methods for optimal precision-recall.

🛡️ Security & Scalability

• —Safe Parsing: Docling runs in a secure, resource-limited container environment.
• —Vector Search Indexing: Optimized for MongoDB Atlas Search, enabling enterprise-grade scaling.
• —Streaming Responses: Uses Server-Sent Events (SSE) logic for smooth user experience.