tomlrogers/esa-multimodal-rag

ESA Multimodal RAG Demo

This project implements a multimodal Retrieval-Augmented Generation (RAG) system capable of answering questions based on text and images extracted from PDF documents.

It uses:

• —Encoder: TigerResearch/VLM2Vec-Qwen2VL-7B for multimodal embeddings (text & images).
• —Generator: mistralai/Mistral-7B-Instruct-v0.2 (loaded in 4-bit) for answer generation, optionally fine-tuned with LoRA.
• —Reranker: BAAI/bge-reranker-large for improving context relevance.
• —Vector Store: FAISS (IndexFlatIP) for efficient similarity search.
• —UI: Streamlit for an interactive web application.

Project Structure

.
├── requirements.txt          # Project dependencies
├── parse_embed.ipynb         # Jupyter notebook for data processing, embedding, and optional LoRA tuning
├── rag_streamlit.py          # Streamlit application for Q&A
├── utils.py                  # Helper functions (parsing, chunking, models, RAG pipeline)
├── data/
│   ├── raw/                  # Place your source PDF files here
│   ├── site_index.faiss      # Generated FAISS index
│   └── meta.jsonl            # Generated metadata for chunks/images
└── models/
    └── mistral-esa-lora/     # Optional LoRA adapter weights (if fine-tuning is performed)

Quick Start

1. 1.Clone the repository:

    git clone <your-repo-url>
    cd <your-repo-name>

1. 1.Set up Environment & Install Dependencies:

    python -m venv venv
    source venv/bin/activate # or venv\Scripts\activate on Windows
    pip install -r requirements.txt

(Note: Installation might take time, especially for PyTorch/CUDA).

1. 1.Add Data: Place your PDF documents into the data/raw/ directory.
2. 2.Process Data: Open and run the parse_embed.ipynb notebook (preferably in VS Code or Jupyter Lab). Execute cells 1, 2, and 3 sequentially. This will:
3. 3.Parse PDFs, extracting text and images.
4. 4.Chunk text and prepare images.
5. 5.Encode text/images into vectors using the Qwen2VL model.
6. 6.Build and save the FAISS index (data/site_index.faiss) and metadata (data/meta.jsonl).
7. 7.(Optional) Fine-tune: If you have a synthetic Q&A dataset (data/synthetic_qa.jsonl), you can enable and run Cell 4 in the notebook to perform LoRA fine-tuning on the Mistral model.
8. 8.Launch App:

    streamlit run rag_streamlit.py

Access the application in your browser (usually at http://localhost:8501).

Usage Notes

• —Adding More PDFs: Simply drop new PDF files into the data/raw directory and rerun Cell 3 (Chunk & Embed) in the parse_embed.ipynb notebook. This will update the FAISS index and metadata file.
• —Performance (Typical on Hugging Face T4 GPU):
• —Embedding Query: ~60 ms
• —FAISS Retrieval: ~5 ms
• —Answer Generation (Mistral-7B): ~3-4 seconds
• —Memory Footprint (4-bit quantization):
• —Qwen2VL-7B Encoder: ~4 GB VRAM
• —Mistral-7B-Instruct Generator: ~4 GB VRAM
• —Reranker + Overhead: ~1-2 GB VRAM
• —(Total fits within a 16 GB T4 GPU)

Deployment (Hugging Face Spaces)

1. 1.Create a new Hugging Face Space.
2. 2.Set the SDK to Streamlit.
3. 3.Choose appropriate hardware (e.g., T4 medium - 16GB GPU).
4. 4.Upload your project files (including the generated data/site_index.faiss and data/meta.jsonl).
5. 5.Ensure requirements.txt is present; Spaces will install dependencies automatically.
6. 6.Use Git LFS for large files like the FAISS index if necessary.
7. 7.The application (rag_streamlit.py) should be set as the entry point.
8. 8.The Space will build and launch the Streamlit app.