Apppublic

RathnavelRahul/omni-modal-rag

sourceHugging Faceupdated 8mo agoView on Hugging Face

0likes

App README

🚀 Omni-Modal RAG System

An intelligent document analysis system powered by AI that can process PDFs, images, and audio files to answer your questions.

Live Demo • Features • Quick Start • Architecture

</div>

🎯 About

Omni-Modal RAG is a Retrieval-Augmented Generation system that enables intelligent document analysis across multiple modalities. Upload documents, images, or audio files, and ask questions in natural language to get accurate, AI-powered answers.

What makes it special?

—Multi-modal support: Process PDFs, images (with OCR), and audio files
—Hybrid search: Combines semantic embeddings + keyword search + knowledge graph
—Real-time streaming: See AI responses as they're generated
—Clean, structured answers: Bullet-point format for easy reading

✨ Features

Feature	Description
📄 PDF Processing	Extract and analyze text from PDF documents
🖼️ Image OCR	Extract text from images using Tesseract
🎵 Audio Transcription	Transcribe audio using OpenAI Whisper
🔍 Hybrid Search	Vector + keyword + knowledge graph retrieval
💬 Streaming Chat	Real-time AI responses with SSE
📊 Knowledge Graph	Entity extraction and relationship mapping
🎨 Modern UI	Beautiful dark-themed React interface

🛠️ Tech Stack

Backend

—FastAPI - High-performance async Python API
—LangChain - LLM orchestration
—FAISS - Vector similarity search
—spaCy - NLP and entity extraction
—Tesseract - OCR for images
—Whisper - Audio transcription
—NetworkX - Knowledge graph

Frontend

—Next.js 14 - React framework
—TypeScript - Type-safe JavaScript
—Tailwind CSS - Utility-first styling
—Lucide Icons - Beautiful icons

AI Models

—Qwen2.5-72B-Instruct - LLM for response generation
—all-MiniLM-L6-v2 - Text embeddings

🏗️ Architecture

┌─────────────────────────────────────────────────────────────────┐
│                        Frontend (Next.js)                        │
│  ┌──────────┐  ┌──────────┐  ┌──────────┐  ┌──────────────────┐ │
│  │  Upload  │  │   Chat   │  │ Sidebar  │  │  Message Display │ │
│  └──────────┘  └──────────┘  └──────────┘  └──────────────────┘ │
└─────────────────────────────┬───────────────────────────────────┘
                              │ HTTP/SSE
                              ▼
┌─────────────────────────────────────────────────────────────────┐
│                      Backend (FastAPI)                           │
│  ┌────────────────────────────────────────────────────────────┐ │
│  │                      API Routes                             │ │
│  │   /upload  │  /chat  │  /files  │  /feedback               │ │
│  └────────────────────────────────────────────────────────────┘ │
│                              │                                   │
│  ┌───────────────────────────┼───────────────────────────────┐  │
│  │                    Processing Pipeline                     │  │
│  │  ┌──────────┐  ┌──────────┐  ┌──────────┐  ┌───────────┐  │  │
│  │  │   PDF    │  │  Image   │  │  Audio   │  │  Chunking │  │  │
│  │  │ Extractor│  │   OCR    │  │ Whisper  │  │  & Index  │  │  │
│  │  └──────────┘  └──────────┘  └──────────┘  └───────────┘  │  │
│  └────────────────────────────────────────────────────────────┘  │
│                              │                                   │
│  ┌───────────────────────────┼───────────────────────────────┐  │
│  │                   Retrieval System                         │  │
│  │  ┌──────────┐  ┌──────────┐  ┌──────────┐  ┌───────────┐  │  │
│  │  │  Vector  │  │ Keyword  │  │Knowledge │  │   HyDE    │  │  │
│  │  │  Store   │  │  Search  │  │  Graph   │  │ Generator │  │  │
│  │  │ (FAISS)  │  │  (BM25)  │  │(NetworkX)│  │           │  │  │
│  │  └──────────┘  └──────────┘  └──────────┘  └───────────┘  │  │
│  └────────────────────────────────────────────────────────────┘  │
│                              │                                   │
│  ┌───────────────────────────┼───────────────────────────────┐  │
│  │                    LLM Integration                         │  │
│  │  ┌────────────────────┐  ┌────────────────────────────┐   │  │
│  │  │  Agent Orchestrator │  │  HuggingFace API Client   │   │  │
│  │  │    (ReAct Loop)     │  │   (Qwen2.5-72B-Instruct)  │   │  │
│  │  └────────────────────┘  └────────────────────────────┘   │  │
│  └────────────────────────────────────────────────────────────┘  │
└─────────────────────────────────────────────────────────────────┘

Data Flow

User Query → Hybrid Retrieval → Context Assembly → LLM Generation → Response
     ↓              ↓                  ↓                ↓              ↓
  "What is X?"   Vector+BM25      Top-K chunks     Qwen2.5-72B    Bullet points
                + Graph Walk      + Related         + Prompt       + Clean format

🚀 Quick Start

Prerequisites

—Python 3.10+
—Node.js 18+
—Git
—Tesseract OCR (for image processing)

1. Clone the Repository

bash

git clone https://github.com/YOUR_USERNAME/omni-modal-rag.git
cd omni-modal-rag

2. Backend Setup

bash

# Navigate to backend
cd backend

# Create virtual environment
python -m venv venv
source venv/bin/activate  # On Windows: venv\Scripts\activate

# Install dependencies
pip install -r requirements.txt

# Download spaCy model
python -m spacy download en_core_web_sm

# Create .env file
cp .env.example .env
# Edit .env and add your HUGGINGFACE_API_KEY

3. Get HuggingFace API Key

1.Go to huggingface.co/settings/tokens
2.Create a new token with "Read" permissions
3.Copy the token to your .env file

4. Frontend Setup

bash

# Navigate to frontend
cd ../frontend

# Install dependencies
npm install

5. Run the Application

Terminal 1 - Backend:

bash

cd backend
python -m uvicorn main:app --reload --port 8000

Terminal 2 - Frontend:

bash

cd frontend
npm run dev

Open http://localhost:3000 in your browser!

🐳 Docker Deployment

Manual Deployment

For detailed instructions on pushing to GitHub and deploying to the cloud (Railway/Render), please see our Deployment Guide.

Using Docker Compose (Local)

bash

# Build and run
docker-compose up --build

# Run in background
docker-compose up -d

Manual Docker Build

bash

# Build backend
docker build -t omni-rag-backend ./backend

# Build frontend
docker build -t omni-rag-frontend ./frontend

# Run containers
docker run -d -p 8000:8000 --env-file .env omni-rag-backend
docker run -d -p 3000:3000 omni-rag-frontend

📡 API Reference

Upload File

http

POST /api/upload
Content-Type: multipart/form-data

file: <binary>

Chat (Streaming)

http

POST /api/chat
Content-Type: application/json

{
  "message": "What are the key findings?",
  "session_id": "optional-session-id"
}

List Files

http

GET /api/files

Delete File

http

DELETE /api/files/{file_id}

⚙️ Configuration

Environment Variables

Variable	Description	Default
`HUGGINGFACE_API_KEY`	HuggingFace API token	Required
`HF_MODEL_ID`	LLM model to use	`Qwen/Qwen2.5-72B-Instruct`
`EMBEDDING_MODEL`	Embedding model	`sentence-transformers/all-MiniLM-L6-v2`
`MAX_FILE_SIZE_MB`	Max file upload size	`10`
`PORT`	Backend port	`8000`

📁 Project Structure

omni-modal-rag/
├── backend/
│   ├── agents/           # LLM orchestration
│   ├── graph/            # Knowledge graph
│   ├── ingestion/        # Document processing
│   ├── llm/              # LLM client
│   ├── retrieval/        # Search & retrieval
│   ├── routes/           # API endpoints
│   ├── synthesis/        # Response building
│   ├── utils/            # Utilities
│   ├── main.py           # FastAPI app
│   ├── config.py         # Configuration
│   └── requirements.txt  # Python dependencies
│
├── frontend/
│   ├── app/              # Next.js pages
│   ├── components/       # React components
│   ├── types/            # TypeScript types
│   └── package.json      # Node dependencies
│
├── docker-compose.yml    # Docker orchestration
├── Dockerfile            # Multi-stage build
└── README.md             # This file

🤝 Contributing

Contributions are welcome! Please feel free to submit a Pull Request.

1.Fork the repository
2.Create your feature branch (git checkout -b feature/AmazingFeature)
3.Commit your changes (git commit -m 'Add some AmazingFeature')
4.Push to the branch (git push origin feature/AmazingFeature)
5.Open a Pull Request

📄 License

This project is licensed under the MIT License - see the LICENSE file for details.

👤 Author

Arman Shalik J

—GitHub: @yourusername
—LinkedIn: Arman Shalik J
—Email: shalik.jhs@gmail.com

⭐ If you found this project helpful, please give it a star! ⭐

</div>