mataX1/siyaq-arabic-rag

🧠 Siyaq AI (سِياق) - Advanced Arabic RAG & Document Parser

An advanced document processing and Retrieval-Augmented Generation (RAG) system tailored specifically to address the unique challenges of the Arabic language—including full diacritics (Harakat) preservation, computational memory optimization, dual-storage integrity, and high-performance semantic retrieval benchmarking.

📧 Contact Information

• —Developer Name: Mathayil Al-Jeraisy
• —Email: Mathayil.alj@gmail.com
• —Phone: +966555257509
• —Demo: [your-live-streamlit-demo-link]

🌟 Features & Architectural Enhancements

Siyaq AI has been engineered and optimized from the ground up to satisfy the highest standards of production-grade AI and document processing systems:

• —[x] Multi-Format Document Parsing: Comprehensive out-of-the-box support for PDF, DOCX, and TXT files. Implements robust, graceful degradation and type safety for unsupported file uploads, entirely preventing system crashes.
• —[x] Shared Embeddings Singleton: An architectural upgrade that loads the massive intfloat/multilingual-e5-base model (~1.1GB) exactly once in RAM, sharing it globally between ChromaDB and the dynamic chunkers. This pattern saves over 1.2GB of RAM, preventing Out-Of-Memory (OOM) crashes in memory-constrained sandbox environments.
• —[x] Multi-Feature Document Analyzer: An intelligent, heuristic-driven classifier that inspects deep document structure (list and bullet ratios, empty-line density, average line lengths, and divider occurrences) to automatically select the optimal chunking strategy:
• —Fixed Chunking: Best for structured lists, tables, and short forms, keeping items contiguous.
• —Dynamic Chunking: Best for continuous paragraphs and narrative prose, maintaining semantic flow.
• —[x] True Dual-Storage Layer:
• —Vector Database (ChromaDB): For top-tier, high-speed semantic similarity searches.
• —Relational Database (SQL - SQLite): Stores unique UUID chunk mappings, parent document relationships, and the actual full text of each chunk, enabling direct SQL queries (search_chunks_sql) and future hybrid keyword lookups.
• —[x] Complete Arabic & Harakat Resilience: Full Arabic language support protecting diacritics (Harakat) from normalization or removal, ensuring high-fidelity semantic representation for embeddings.
• —[x] Comprehensive Retrieval Benchmark Suite: A scientific evaluation framework (run_benchmark.py) assessing retrieval performance on classic RAG criteria:
• —Hit Rate, Recall, and Mean Reciprocal Rank (MRR).
• —Precise retrieval latency tracking in milliseconds.
• —Diacritics Resilience: Automatically tests plain-text queries against fully diacritized documents to verify semantic alignment.
• —Automatically exports comprehensive reports directly to a dedicated markdown file: benchmark/benchmark_results.md.
• —[x] Complete Automated Pytest Suite: 100% automated coverage for all critical components (Parsers, Chunkers, Arabic Processor, Databases, and the RAG Pipeline), guaranteeing extreme maintainability and code robustness.

🏗️ System Architecture

The codebase follows clean, decoupled, and highly modular design principles:

1. 1.Parser Factory (`app/parser`): Abstracts raw file parsing, leveraging PyMuPDF (optimized with custom Unicode CMap remapping and system control byte sanitization to eliminate boxes or spacing issues) and python-docx.
2. 2.Arabic Processor (`app/arabic`): Handles Arabic text cleaning, whitespace normalization, and encoding alignments while strictly protecting diacritics.
3. 3.Smart Chunker (`app/chunker`): Employs the DocumentAnalyzer to dynamically route documents to FixedChunker or DynamicChunker depending on text structure.
4. 4.Dual Storage Layer (`app/storage`):
5. 5.embeddings.py (Singleton): Reuses the shared embedding models.
6. 6.VectorStore: Powers vector storage and semantic queries via ChromaDB.
7. 7.SQLStore: Manages metadata and full-text databases via SQLite.
8. 8.RAG Pipeline (`app/rag`): Orchestrates document ingestions, executes queries against databases, and links with Google Gemini or Groq to synthesize context-grounded answers.

🛠️ How to Run & Verify

🚀 Quick Start: Run via Self-Healing Script (Recommended)

We have provided a self-healing startup script that automatically creates the virtual environment, installs/repairs missing dependencies, and safely runs the Streamlit server:

chmod +x run.sh
./run.sh

📦 Manual Setup:

If you prefer to configure the environment manually:

python3 -m venv venv
source venv/bin/activate
pip install -r requirements.txt

2. Configure Gemini API Key:

Create a .env file in the root directory:

GOOGLE_API_KEY=your_gemini_api_key_here

(Alternatively, you can securely enter your key dynamically via the Streamlit Sidebar during runtime).

3. Run Automated Tests (Pytest):

To verify the integrity of all codebase modules:

pytest tests/ -v

4. Run Retrieval Benchmarks:

To measure retrieval accuracy, diacritics resilience, latency, and memory footprint:

python benchmark/run_benchmark.py

This prints a detailed metrics summary and saves a permanent report to: `benchmark/benchmark_results.md`.

5. Launch the Streamlit Web Application:

streamlit run app.py

💡 Future Roadmap

1. 1.Graph RAG: Implement dynamic entity-relation graphs using LLMs to trace key concepts and link them across multiple documents.
2. 2.BM25 Hybrid Search: Blend dense vector search with sparse keyword-based BM25 indexes to guarantee keyword-perfect lookups.
3. 3.Re-ranking (Cohere/BGE): Integrate a second-stage cross-encoder model to re-rank retrieved contexts, maximizing RAG accuracy.