LuisMBA/multimodal_RAG_kaggle_based

1---2title: Multimodal RAG Kaggle Based3emoji: 👁4colorFrom: red5colorTo: pink6sdk: gradio7sdk_version: 5.25.18app_file: app.py9pinned: false10license: apache-2.011short_description: Multimodal RAG to augment english recipes searches12---13 14# Multimodal Retrieval System with FAISS15 16This repository contains a prototype system for multimodal information retrieval using FAISS, capable of searching across text and images using vector similarity.17 18## Structure19 20- `notebook/` (or `.ipynb`): Contains the logic to generate the vector indexes for both text and images.21- `app.py`: Gradio-based interface for interacting with the system.22- `search_ocean.py`: Core logic for performing FAISS-based similarity search using precomputed indexes.23- `text_index.faiss`, `image_index.faiss`: The FAISS index files generated by the notebook (already included in the app).24- `metadata_text.json`, `metadata_image.json`: Associated metadata for mapping index results back to source information.25 26## What it does27 28- Loads precomputed FAISS indexes (for text and image).29- Performs retrieval based on a text or image query.30- Returns top matching results using cosine similarity.31 32## What it doesn't (yet) do33 34- No generation step (e.g., using LLMs) is implemented in this app.35- While the code for image retrieval is ready, image indexes must be built in the notebook beforehand.36- There is **no context overlap** implemented when chunking the data for indexing. Each chunk is indexed independently, which may affect the quality of retrieval in some use cases.37 38## Dependencies39 40- `faiss-cpu`41- `sentence-transformers`42- `openai-clip`43- `torch`44- `torchvision`45- `gradio`46- `Pillow`47 48## Notes49 50- The app is designed to separate concerns between indexing (offline, notebook) and retrieval (live, Gradio app).51- You can easily extend this to include LLM generation or contextual QA once relevant results are retrieved.52