aarushi63/scholarship-rag

🎓 ScholarMatch AI

An AI-powered scholarship discovery and recommendation platform for Indian students.

ScholarMatch AI uses Retrieval-Augmented Generation (RAG) to match students with relevant government scholarship schemes based on their profile — caste category, state, income, and education level. An integrated AI chat advisor powered by Google Gemini answers student queries with cited, grounded responses.

Dataset: 45 government scholarship schemes · 180 text chunks · Hybrid retrieval latency ~200 ms · Recall@5 43% on a 6-example golden set

✨ Features

• —Hybrid Search — Combines dense semantic embeddings (BAAI/bge-small-en-v1.5) with sparse TF-IDF vectors in Qdrant for accurate retrieval.
• —AI Chat Advisor — Gemini 2.5 Flash generates grounded answers, linking to specific scheme pages.
• —Personalised Recommendations — Schemes are ranked by an eligibility scorer that weighs the user's category, state, income, and education level.
• —User Accounts — JWT-based auth with profile management (category, state, income, education level).
• —Saved Schemes — Users can bookmark and manage their favourite scholarships.
• —Dashboard — At-a-glance stats and personalised scheme highlights.
• —PDF Ingestion — Parse government scholarship notification PDFs to automatically extract structured scheme data.
• —RAGAS Evaluation — Built-in RAG evaluation pipeline (eval/) to measure retrieval quality.

🗂️ Project Structure

scholarship_Rag/
├── backend/                # FastAPI REST API
│   ├── auth/               # JWT authentication
│   ├── db/                 # SQLAlchemy models, Postgres & Qdrant clients
│   ├── recommendation/     # Eligibility scoring & ranking logic
│   ├── routes/             # API route handlers (chat, schemes, profile, saved, dashboard)
│   ├── config.py           # Pydantic settings (reads from .env)
│   ├── schemas.py          # Pydantic request/response schemas
│   └── main.py             # FastAPI app entry point
├── frontend/               # Next.js 16 + TypeScript frontend
│   ├── app/                # App Router pages
│   └── lib/                # Shared utilities & API client
├── ingestion/              # Data pipeline: chunk → embed → index into Qdrant
│   ├── chunker.py          # Splits scheme text into overlapping chunks
│   ├── embedder.py         # Dense + sparse embedding generation
│   ├── index_data.py       # CLI to index data/sample_schemes.json into Qdrant
│   └── schema.py           # SchemeMetadata Pydantic model
├── scraper/                # PDF & web scrapers
│   ├── pdf_parser.py       # Parse scholarship notification PDFs
│   └── sources/            # Source-specific scrapers
├── data/                   # Scholarship dataset (JSON) & TF-IDF vectorizer
├── eval/                   # RAGAS-based RAG evaluation scripts
├── scripts/                # Helper scripts
├── docker-compose.yml      # Local Qdrant + Redis + Postgres stack
├── render.yaml             # Render.com deployment config (backend)
└── requirements.txt        # Python dependencies

🚀 Getting Started

Prerequisites

1. Clone the Repository

git clone https://github.com/aarushi2810/scholarship_Rag.git
cd scholarship_Rag

2. Configure Environment Variables

cp .env.example .env

Edit .env and fill in your keys:

Note: For local development you can run the full database stack with Docker (see step 3) and set DATABASE_URL to postgresql+asyncpg://postgres:postgres@localhost:5432/scholarshiprag.

3. Start Local Services (Docker)

docker compose up -d

This starts Qdrant (port 6333), Redis (port 6379), and PostgreSQL (port 5432).

4. Set Up Python Backend

python -m venv .venv
source .venv/bin/activate          # Windows: .venv\Scripts\activate
pip install -r requirements.txt

5. Index the Scholarship Data

python -m ingestion.index_data

This reads data/sample_schemes.json, generates dense + sparse embeddings, and uploads them to your Qdrant collection.

6. Run the Backend

uvicorn backend.main:app --reload --port 8000

API docs available at http://localhost:8000/docs.

7. Run the Frontend

cd frontend
npm install
npm run dev

Frontend available at http://localhost:3000.

🔌 API Overview

🧠 How the RAG Pipeline Works

flowchart TD
    A["User message"] --> B["Intent detection"]
    B --> C["Entity extraction\n(state, category, income, education)"] 
    C --> D["Query rewriting\n(vague queries only)"] 
    D --> E["Hybrid search — Qdrant\nDense: BGE-small cosine + Sparse: TF-IDF BM25\ntop-20 candidates"]
    E --> F["FlashRank cross-encoder reranking\ntop-7 passages"]
    F --> G["Eligibility scoring\nweighted match: category · state · education · deadline"]
    G --> H["Gemini 2.5 Flash\nsystem prompt tailored to intent\nretry + exponential backoff"]
    H --> I["Response formatting\nemoji strip · citation injection · 600-word cap"]
    I --> J["Chat response + source citations"]

    subgraph Caches
        K["Embedding cache\nTTL 30 min"]
        L["Result cache\nTTL 5 min"]
    end
    E --> K
    E --> L

📄 Ingesting New PDFs

Use the built-in PDF parser to extract scholarship data from government notification PDFs:

# Parse a single PDF
python -m scraper.pdf_parser data/raw/pdfs/nsp_notification.pdf

# Parse all PDFs in a directory
python -m scraper.pdf_parser data/raw/pdfs/

The parser extracts: scheme name, eligibility text, income ceiling, deadline, benefits, documents required, caste categories, and education levels — all via regex heuristics tuned for government document formats.

📊 Running RAG Evaluation

The eval/ directory contains retrieval evaluation scripts measuring recall and latency.

# Run the hybrid retrieval benchmark (dense vs hybrid comparison)
python -m eval.run_retrieval_eval

# Inspect raw passage payloads from Qdrant
python eval/scratch/inspect_payloads.py

Results on the 6-example golden set: | Metric | Dense only | Hybrid | |--------|-----------|--------| | Recall@5 | 40.5% | 43.0% | | Latency (p50) | ~337 ms | ~203 ms |

🐳 Docker Deployment

The backend ships with a Dockerfile and the project includes a render.yaml for one-click deployment to Render.com:

# Build and run backend container locally
docker build -f backend/Dockerfile -t scholarmatch-backend .
docker run -p 8000:8000 --env-file .env scholarmatch-backend

🛠️ Tech Stack

⚠️ Known Limitations

• —Dataset coverage: 45 government schemes covering 9 Indian states + All-India schemes. Private scholarships and institution-specific awards are not included.
• —Tamil Nadu and some other states have no state-specific schemes in the dataset; queries default correctly to All-India scholarships.
• —Gemini dependency: Chat requires a valid GEMINI_API_KEY. If unavailable, the advisor falls back to structured plain-text answers from retrieved passages.
• —English only: Queries must be in English; regional language support is not implemented.
• —Income data: Some older schemes have no income ceiling in the dataset; eligibility scoring treats them as income-unrestricted.

🤝 Contributing

1. 1.Fork the repository
2. 2.Create a feature branch: git checkout -b feature/my-feature
3. 3.Commit your changes: git commit -m "Add my feature"
4. 4.Push the branch: git push origin feature/my-feature
5. 5.Open a Pull Request

📝 License

This project is licensed under the MIT License.