CGIAR/ifpri-dp-rag

IFPRI Discussion Papers — AI-Powered Search

Semantic search and Q&A over 1,681 IFPRI Discussion Papers using a lightweight RAG (Retrieval-Augmented Generation) pipeline.

How it works

1. 1.Embeddings — All papers are chunked and embedded with `BAAI/bge-small-en-v1.5` (33 M parameters, CPU-friendly).
2. 2.Vector search — A pre-built FAISS index retrieves the most relevant chunks for each query.
3. 3.LLM answer — `mistralai/Mistral-7B-Instruct-v0.3` (7 B) generates a concise, cited answer via the HF Inference API.

Running locally

1 — Install dependencies

pip install -r requirements.txt

2 — Build the vector index (one-time, ~1–2 h on CPU)

# from the ifpri-rag-app/ directory
python ingest.py --pdf_dir ../IFPRI-DiscussionPapers --output_dir .

This produces faiss_index/ and metadata.json.

3 — Set your Hugging Face token

export HF_TOKEN=hf_...

Get a free token at <https://huggingface.co/settings/tokens>. The token is used to call the Mistral inference endpoint (free tier).

4 — Launch the app

python app.py

Open <http://localhost:7860> in your browser.

Deploying to CGIAR's Hugging Face Space

a) Create the Space

1. 1.Go to <https://huggingface.co/cgiar> (or the relevant org).
2. 2.Click New Space → SDK: Gradio → name it e.g. ifpri-dp-search.

b) Upload files

Upload these files to the Space repository:

app.py
requirements.txt
README.md
faiss_index/            ← generated by ingest.py
metadata.json           ← generated by ingest.py

Tip: If faiss_index/ is large (typically 200–500 MB), use git lfs or upload via the HF web UI.

c) Add the secret token

In the Space settings → Secrets → add:

d) Optional environment variables

Project structure

ifpri-rag-app/
├── app.py          # Gradio application
├── ingest.py       # PDF → FAISS index builder (run locally)
├── requirements.txt
├── README.md       # This file (also the HF Space card)
├── faiss_index/    # ← generated; upload to Space
└── metadata.json   # ← generated; upload to Space

License

MIT — research and educational use. IFPRI Discussion Papers are © IFPRI and subject to their respective licenses.