Apppublic

annyss/palestine-smart-library

sourceHugging Faceupdated 5mo agoView on Hugging Face

0likes

App README

🇵🇸 Palestine Smart Library — AI for Palestine Competition

A RAG-based (Retrieval-Augmented Generation) multilingual chatbot answering questions on the Palestinian cause from 15 official documents, with zero hallucination.

Features

Feature	Details
Bilingual RAG Chat	Arabic & English, auto-detected, same language response
Anti-Hallucination	Cosine similarity threshold + strict system prompt
Citations	Every answer cites `[Document Title, Page X]`
Upload PDF	Index new documents instantly without restart
Discourse Analysis	Keyword/entity frequency + AI thematic analysis
Compare Documents	Side-by-side LLM comparison of two documents
Document Summary	Per-document AI summaries
Interactive Map	Folium map of Palestinian cities, camps, crossings
Historical Timeline	Plotly timeline from 1897 to 2025
Word Cloud	Arabic & English word clouds per document
Statistics	Charts: pages, word counts, language distribution
Export Chat	Download full conversation as text
Auto-translate	Translate answers to the opposite language

Architecture

User Query (Arabic or English)
        │
        ▼
Language Detection (langdetect)
        │
        ▼
Query Embedding: "query: {text}"
   [multilingual-e5-large]
        │
        ▼
FAISS IndexFlatIP (cosine similarity)
   top-k=5 chunks retrieved
        │
        ▼
Similarity Threshold ≥ 0.35?
   NO  → "Not found in documents"
   YES → Continue
        │
        ▼
LLM (Qwen3-30B-A3B-Thinking via OpenAI-compatible API)
   Strict system prompt — no external knowledge
        │
        ▼
Answer + Citations [Title, Page X]
  (in same language as query)

Tech Stack

Component	Technology
LLM	Qwen3-30B-A3B-Thinking (configurable)
Embeddings	`intfloat/multilingual-e5-large`
Vector DB	FAISS (IndexFlatIP, cosine similarity)
PDF Parsing	PyMuPDF (fitz) — Arabic RTL support
Language Detection	langdetect
UI	Gradio 4.x
Maps	Folium + HeatMap
Charts	Plotly
Word Clouds	WordCloud library

Quick Start

1. Clone and configure

bash

git clone <repo>
cd coderag
cp .env.example .env
# Edit .env — add your API key and model endpoint

2. Install dependencies

bash

pip install -r requirements.txt

3. Build the FAISS index (run once)

bash

python build_index.py
# Takes ~5-15 min first time (downloads multilingual-e5-large + embeds 15 PDFs)
# Saves index/faiss.index + index/metadata.pkl for instant reload

4. Run the app

bash

python app.py
# Open http://localhost:7860

Environment Variables (.env)

env

LLM_PROVIDER=openai_compatible       # "groq" or "openai_compatible"
OPENAI_COMPATIBLE_KEY=sk-...         # API key
OPENAI_COMPATIBLE_BASE_URL=https://api.together.xyz/v1
OPENAI_COMPATIBLE_MODEL=Qwen/Qwen3-30B-A3B-Thinking
GROQ_API_KEY=gsk_...                 # Only needed if LLM_PROVIDER=groq
EMBEDDING_MODEL=intfloat/multilingual-e5-large

Document Corpus (15 PDFs)

English (8)

—UN General Assembly Resolution 194 (1948)
—UNRWA Technical Report 2024
—Humanitarian Situation Update 176 – Gaza Strip (OCHA)
—Gaza Update Report – November 2024 (OPT)
—Israel-Palestine History Timeline 2024–25
—Palestinian Historical Calendar
—Palestinian Identity – Rashid Khalidi
—The Hundred Years' War on Palestine

Arabic (7)

—تقرير غزة الإنساني 2024
—ذاكرة المكان
—شخصيات فلسطينية
—فلسطين العربية
—فلسطين
—كتاب النخبة – الجزء الأول
—كتاب النخبة – الجزء الثاني

Anti-Hallucination Design

1.Vector similarity threshold: Best retrieved chunk cosine similarity < 0.35 → system returns the exact string "This information was not found in the available documents." — LLM is never called.
2.Strict system prompt: LLM instructed to answer only from provided excerpts, cite sources, and not use external knowledge.
3.Temperature = 0.1: Minimal creativity, maximum factuality.
4.No web search: Completely offline retrieval (no external APIs during inference).

Project Structure

coderag/
├── app.py                  # Main Gradio app
├── build_index.py          # One-time index builder
├── requirements.txt
├── .env                    # API keys (never commit)
├── .env.example
├── official_docs/          # 15 PDFs
├── index/                  # FAISS index (built by build_index.py)
├── rag/
│   ├── document_loader.py  # PDF loading + chunking + metadata
│   ├── embeddings.py       # multilingual-e5-large wrapper
│   ├── vector_store.py     # FAISS wrapper with dynamic add
│   └── generator.py        # LLM generation + anti-hallucination
├── tabs/
│   ├── chat_tab.py
│   ├── discourse_tab.py
│   ├── compare_tab.py
│   ├── summary_tab.py
│   ├── map_tab.py
│   ├── timeline_tab.py
│   ├── wordcloud_tab.py
│   ├── statistics_tab.py
│   ├── upload_tab.py
│   └── about_tab.py
└── utils/
    └── language.py         # Language detection

Competition

AI for Palestine — Smart Library Submission deadline: April 24, 2026 Team email: anisbensab@gmail.com