Apppublic

mukimshardul/inverse_problem

sourceHugging Faceupdated 7mo agoView on Hugging Face

0likes

App README

🔬 Multimodal RAG System for Scientific Papers

A production-grade multimodal Retrieval-Augmented Generation (RAG) system built to query research papers by their figures, plots, and tables — not just text. It uses deep learning layout detection, tri-modal hybrid search, and a local Vision Language Model (VLM) to synthesize scientific answers.

⚡ Quick Start

bash

# Step 1 – Activate environment
conda activate rag_prod

# Step 2 – Run the app (if already indexed)
streamlit run app.py

App runs at: http://localhost:8501

🗂️ Project Structure

NLP/
├── raw/                        # Input PDFs go here
├── extracted_data/
│   ├── images/                 # Cropped figures/tables (PNG)
│   ├── images_metadata_d2.json # V1 extraction metadata
│   └── images_metadata_v2.json # V2 VLM-enriched metadata (generated by enrich_metadata.py)
├── index/
│   ├── chroma_db/              # ChromaDB — visual + semantic vector stores
│   └── bm25_index.pkl          # BM25 lexical index
├── src/
│   ├── extract_detectron2.py   # Step 1: Extract figures using Detectron2
│   ├── build_index.py          # Step 2: Embed & index all figures
│   ├── enrich_metadata.py      # Optional: VLM enrichment of captions
│   └── retriever.py            # Tri-Modal Hybrid Retriever (standalone)
├── v1_deployment/              # Self-contained V1.0 scripts (commented)
├── model_final.pth             # PubLayNet Detectron2 weights
├── app.py                      # Main Streamlit Application
└── requirements.txt

🔄 Full Pipeline

Step 1 — Extract Figures from PDFs

bash

conda run -n rag_prod python src/extract_detectron2.py

—Reads all .pdf files in raw/
—Uses layoutparser + Detectron2 (mask_rcnn_X_101_32x8d_FPN_3x, PubLayNet) to detect Figure and Table regions
—Adds a 20-pixel padding on bounding boxes to preserve plot axis labels
—Extracts spatially-proximate text (captions) for each detected region
—Saves cropped images → extracted_data/images/
—Saves metadata → extracted_data/images_metadata_d2.json

Step 2 (Optional) — Enrich Captions via VLM

bash

# Runs in background — 221 images × ~10s each
nohup conda run -n rag_prod python src/enrich_metadata.py > enrich.log 2>&1 &

—Sends each extracted plot to Llama 3.2 Vision via Ollama
—Generates a detailed scientific description of each figure
—Saves enriched metadata → extracted_data/images_metadata_v2.json
—Indexer automatically picks this up if it exists

Step 3 — Build the Search Index

bash

conda run -n rag_prod python src/build_index.py

Populates two ChromaDB collections and one BM25 index:

Index	Model	Purpose
`multimodal_figures` (ChromaDB)	OpenCLIP `ViT-B-32`	Visual similarity
`semantic_texts` (ChromaDB)	MiniLM `all-MiniLM-L6-v2`	Semantic caption similarity
`bm25_index.pkl`	BM25Okapi	Keyword matching

Step 4 — Launch the App

bash

conda run -n rag_prod streamlit run app.py --server.port 8501

🔍 How Retrieval Works

The system uses Tri-Modal Reciprocal Rank Fusion (RRF):

Query Text
    │
    ├──► OpenCLIP ──► Visual embedding ──► ChromaDB (multimodal_figures) ──► Rank list A
    │
    ├──► MiniLM ────► Text embedding ───► ChromaDB (semantic_texts)      ──► Rank list B
    │
    └──► BM25 ──────► Tokenized words ──► BM25 Index                     ──► Rank list C
                                                     │
                                    ╔════════════════╩═══════════════╗
                                    ║  Reciprocal Rank Fusion (RRF)  ║
                                    ║  score = Σ weight / (60 + rank)║
                                    ╚════════════════════════════════╝
                                                     │
                                            Top-K Merged Results

The 3 weights are fully tunable in the UI sidebar in real-time.

🧠 VLM Answer Generation

After retrieval, the top-K figures are:

1.Base64-encoded and batched
2.Combined with their captions as text context
3.Sent to Llama 3.2 Vision running locally via Ollama
4.The answer is streamed token-by-token back to the UI

📦 Dependencies

bash

conda install -c conda-forge layoutparser detectron2 sentence-transformers
pip install open_clip_torch chromadb rank_bm25 streamlit ollama pymupdf

Full pinned list: requirements.txt

🔑 Key Design Decisions

Decision	Why
Detectron2 over PyMuPDF for extraction	Detectron2 finds figures semantically, not just embedded image blobs
20px bounding box padding	Axis labels, tick marks, and legends are just outside the detected box
OpenCLIP `ViT-B-32` (LAION)	Strong general CLIP model, good for visual image-text alignment
MiniLM `all-MiniLM-L6-v2`	Fast & accurate semantic text matching for scientific captions
RRF instead of score blending	Different models have incompatible score scales; RRF is rank-safe
Local Llama 3.2 Vision via Ollama	Privacy-preserving, no API cost, multimodal
VLM caption enrichment at index-time	Scientific plots rarely have good captions; VLM writes better ones

📁 V1.0 Standalone (`v1_deployment/`)

A clean, fully-commented, dependency-isolated copy of the core pipeline scripts. Each script can be run independently without the src/ module structure.

bash

cd v1_deployment/
python extract_images.py        # Extract
python build_index.py           # Index
streamlit run retriever_app.py  # Run

📝 Notes

—The VLM enrichment script (enrich_metadata.py) may run for 30–60 minutes given 221 figures. It saves progress every 10 items, so it can be safely interrupted and restarted.
—After enrichment finishes, re-run build_index.py to incorporate the richer captions into the vector store.
—The Streamlit app will automatically use images_metadata_v2.json (enriched) if present, otherwise falls back to images_metadata_d2.json.

🔬 Multimodal RAG System for Scientific Papers

⚡ Quick Start

🗂️ Project Structure

🔄 Full Pipeline

Step 1 — Extract Figures from PDFs

Step 2 (Optional) — Enrich Captions via VLM

Step 3 — Build the Search Index

Step 4 — Launch the App

🔍 How Retrieval Works

🧠 VLM Answer Generation

📦 Dependencies

🔑 Key Design Decisions

📁 V1.0 Standalone (v1_deployment/)

📝 Notes

📁 V1.0 Standalone (`v1_deployment/`)