Usmanbhat/fiqh-ai-api

🚀 Fiqh.ai FastAPI Retrieval & Hybrid Search API

This repository contains the containerized FastAPI backend for Fiqh.ai—a citation-safe, high-performance hybrid reference-search engine specialized in classical Hanafi Fiqh texts.

It is designed to run in a cloud-connected environment inside a Docker container on Hugging Face Spaces, serving as the AI retrieval core for the Next.js web application.

⚡ Key Architectural Features

• —Dual Search Engine (Lexical & Semantic):
• —Lexical Matching: Powered by SQLite FTS5 (BM25) with custom Arabic diacritic-insensitive normalization filters.
• —Semantic AI Search: Powered by a SentenceTransformers model (asafaya/bert-base-arabic) translating search queries into 768-dimensional sentence vectors.
• —Reciprocal Rank Fusion (RRF): Mathematically blends exact lexical matches and concept-based vector matches using rank-order reciprocal summing, prioritizing results that satisfy both conditions without numerical metric distortion.
• —Pristine Highlight Engine: Leverages orthographic regex to identify matching Arabic phrases (handling standard variations of Alif, Yaa, Taa-Marbutah, and complex multi-layered harakat/diacritics) to inject clean HTML highlights on the fly.
• —Scholarly Surrounding Context: Provides dynamic endpoint retrievals (/chunks/{chunk_id}/context) to pull surrounding paragraphs/pages (before and after a target passage) keeping studies fluid and coherent.

🛡️ API Endpoints

1. GET /health

Returns database connectivity status and local file path configurations.

2. GET /books

Retrieves a complete dictionary of all indexed books, metadata, and chunk statistics in the database.

3. GET /search

The primary retrieval gateway supporting three search modes:

• —Query Parameters:
• —q: Search keyword or phrase.
• —limit: Results count limit (defaults to 10, max 50).
• —book_ids: Comma-separated list of IDs to filter search results by specific books.
• —mode: Match strictness: all (AND logical), any (OR logical), or exact (exact phrase).
• —Search Routing Logic:
• —Checks if chunk_embeddings table exists and contains data.
• —If disabled/empty: Gracefully executes the lexical BM25 database search.
• —If active/populated: Automatically encodes the query into a vector, queries candidate embeddings, computes cosine similarities in Python, and blends them using the RRF algorithm.

4. GET /chunks/{chunk_id}

Retrieves a specific classical segment by its row ID.

5. GET /chunks/{chunk_id}/context

Retrieves the target passage alongside its immediate surrounding pages (using a configurable sliding window) for complete context readout.

🗃️ Database Layout & Semantic Indexing

The underlying database uses SQLite3 with WAL mode enabled.

To enable AI semantic search, we use a custom table chunk_embeddings storing serialized vector blobs generated by sentence-transformers:

CREATE TABLE chunk_embeddings (
    chunk_id INTEGER PRIMARY KEY REFERENCES chunks(id),
    embedding BLOB NOT NULL
);

Running Local Semantic Vector Generation

To generate the semantic embeddings locally using your machine's CPU/GPU and upload the populated database:

# 1. Install dependencies
pip install sentence-transformers torch numpy

# 2. Decompress database
gunzip -k data/index/fiqh.db.gz

# 3. Generate embeddings (automatically utilizes Apple Silicon MPS or CUDA GPU)
python3 scripts/generate_embeddings.py

# 4. Re-compress and push to Hugging Face
gzip -f data/index/fiqh.db
git add data/index/fiqh.db.gz
git commit -m "Upload fully populated AI vector database"
git push origin main

🐳 Docker Deployment Setup

Hugging Face Spaces builds and runs this API inside a secure non-root Docker environment using the following pipeline (Dockerfile):

1. 1.Base: python:3.9-slim with system utilities (build-essential, git).
2. 2.Environment: Standard user UID 1000 to comply with Hugging Face Space security protocols.
3. 3.Extraction: Automated gunzip decompressing of fiqh.db.gz to ensure zero cold-start overhead or disk waste.
4. 4.Port: Standard Hugging Face port 7860.