Prateek-Dhar-Dwivedi/AM-RAG-ASAE-Grading
0
AM-RAG-ASAE: Adaptive Multilingual RAG-Based Explainable Automatic Short Answer Evaluation for Personalized Learning
   
📌 1. Project Overview & Research Contribution
AM-RAG-ASAE (Adaptive Multilingual RAG-Based Explainable Automatic Short Answer Evaluation for Personalized Learning) is an advanced NLP and retrieval-augmented assessment framework designed for automated short-answer evaluation (ASAG).
Unlike conventional semantic-similarity or naive LLM-prompting techniques, AM-RAG-ASAE incorporates:
- Multi-Signal Two-Stage Retrieval: FAISS Top-50 candidate retrieval followed by a 5-signal reranker optimizing answer semantic similarity ($S{\text{ans}}$), question relevance ($S{\text{q}}$), concept coverage ($S{\text{concept}}$), reference-answer similarity ($S{\text{ref}}$), and lexical overlap ($S_{\text{lex}}$).
- Three Isolated Vector Collections:
- Collection A: Expert-Graded Training Exemplars (zero test-set leakage).
- Collection B: Educational Knowledge Textbook Passages.
- Collection C: Diagnostic Scientific Misconceptions Catalog.
- Misconception-Aware Diagnostic Grading: Rigorous separation between Missing Concepts (omitted facts) and Alternative Misconceptions (flawed mental models).
- Pedagogical Explainability & Personalized Remediation: Actionable formative feedback citing exact textual evidence and generating 4-part personalized study modules (Topic, Conceptual Review, Worked Exemplar, Formative Practice Question).
- Cross-Lingual Multilingual Extension: Native support for English, Hindi, and Punjabi.
[!NOTE] Architecture Clarification: This framework strictly employs Vector RAG (Sentence Embeddings + FAISS Flat Inner-Product Indexing + Multi-Signal Reranking + LLM Synthesis). It does NOT use GraphRAG or Knowledge Graphs.
🏗️ 2. Overall Architecture
USER (Student / Educator)
│
▼
┌───────────────────────┐
│ Question + Answer │
└───────────┬───────────┘
│
▼
LANGUAGE DETECTION
(EN / HI / PA Detection)
│
▼
TEXT PREPROCESSING
(Unicode, Whitespace, Punct)
│
┌──────────────┴──────────────┐
▼ ▼
QUESTION ANALYSIS ANSWER ANALYSIS
│ │
▼ ▼
Question Embedding Answer Embedding
│ │
│ Concept Extraction
│ (spaCy / Keyphrase)
│ │
└──────────────┬──────────────┘
▼
FAISS VECTOR SEARCH
(Candidate Top-50 Retrieval)
│
▼
MULTI-SIGNAL RERANKER
┌──────────────────────┼──────────────────────┐
▼ ▼ ▼
Answer Similarity Question Relevance Concept Coverage
(S_answer) (S_question) (S_concept)
│ │ │
├──────────────────────┼──────────────────────┤
▼ ▼
Reference Similarity Lexical Similarity
(S_reference) (S_lexical)
│ │
└──────────────────────┬──────────────────────┘
▼
TOP-K
│
┌──────────────────────┼──────────────────────┐
▼ ▼ ▼
Knowledge Passages Expert Exemplars Misconceptions
(Collection B) (Collection A) (Collection C)
│ │ │
└──────────────────────┼──────────────────────┘
▼
EVIDENCE BUILDER
│
▼
LLM GRADER
│
┌──────────────────────┼──────────────────────┐
▼ ▼ ▼
Score / Grade Concept Analysis Misconceptions
│ │ │
└──────────────────────┼──────────────────────┘
▼
MISSING CONCEPTS
│
▼
EXPLAINABLE FEEDBACK
│
▼
PERSONALIZED RECOMMENDER
(Topic, Review, Example, Practice)
│
▼
FINAL STRUCTURED JSON📊 3. Empirical Research Results & Baseline Comparison
All experiments were evaluated on an unseen, stratified 15% test set of the SciEntsBank benchmark. To ensure complete integrity, the test set was never indexed in FAISS or used for hyperparameter tuning.
Experiment 1: Baseline Comparison Table
Experiment 4: Top-K Sensitivity Curve
Experiment 6: Component Ablation Study
Experiment 9: Multilingual Evaluation (English / Hindi / Punjabi)
💻 4. Installation & Environment Setup
All files, caches, and models reside strictly in E:\NLP_Paper_Imple.
1. Clone or Navigate to Project Directory
cd E:\NLP_Paper_Imple\AM-RAG-ASAE2. Activate Virtual Environment
E:\NLP_Paper_Imple\venv\Scripts\Activate.ps13. Verify Dependencies
Dependencies installed in requirements.txt:
torch,transformers,sentence-transformers,faiss-cpu,spacy,nltk,scikit-learn,fastapi,uvicorn,streamlit,langdetect,rank-bm25,matplotlib,seaborn.
🚀 5. Command-Line Interface (CLI) Usage
The unified CLI run.py supports all lifecycle operations:
Data Preparation & Preprocessing
python run.py --mode prepare-dataFAISS Vector Indexing (Zero Leakage)
python run.py --mode build-indexValidation Weight Optimization
python run.py --mode trainRun Full Research Experiments Suite (Exp 1 - 9)
python run.py --mode experimentsEvaluate a Single Input Response
python run.py --mode evaluate --question "Explain how plants make their own food through photosynthesis." --student_answer "Plants absorb sunlight using chlorophyll and combine carbon dioxide and water to produce glucose sugar while releasing oxygen."Run Automated Unit Tests
python run.py --mode test🌐 6. REST API & Web Dashboard
1. Launch FastAPI Backend
python run.py --mode apiAccess interactive OpenAPI documentation at: http://127.0.0.1:8000/docs
Core Endpoints:
GET /health: Service status and collection telemetry.POST /evaluate: Full multi-signal grading, misconception analysis, and recommendations.POST /retrieve: Top-K evidence retrieval with individual signal breakdowns.POST /feedback: Formative pedagogical feedback generation.POST /recommend: 4-part personalized study modules.
2. Launch Interactive Streamlit UI
streamlit run frontend/app.pyUI Features (4 Pages):
- Student Evaluation: Interactive input form with benchmark presets and real-time grading.
- Evaluation Results: Visual score dials, present vs. missing concept tags ($\checkmark / \triangle$), misconception diagnostic alerts ($\times$), and personalized study cards.
- Retrieval Explainability: 5-signal breakdown table ($S{\text{ans}}, S{\text{q}}, S{\text{concept}}, S{\text{ref}}, S_{\text{lex}}$) and interactive composite score chart.
- Research Dashboard: Publication-ready benchmark tables, Top-K sensitivity curves, ablation bar charts, and multilingual cross-lingual metrics.
📁 7. Project File Structure
E:\NLP_Paper_Imple\AM-RAG-ASAE\
├── api\
│ └── main.py # FastAPI REST server
├── baselines\
│ └── baseline.py # 6 Benchmark Baselines (TF-IDF, SBERT, LLM, Standard RAG)
├── configs\
│ └── config.yaml # System configurations, models, weights, paths
├── data\
│ ├── data_loader.py # SciEntsBank dataset builder & 70/15/15 stratified splitter
│ ├── domains.py # Domain taxonomies, Knowledge & Misconceptions corpus
│ ├── train\train.json # Training split (Indexed in FAISS)
│ ├── validation\val.json # Validation split (Used for weight tuning)
│ ├── test\test.json # Unseen Test split (Strictly evaluated once)
│ └── knowledge\ # Educational knowledge and misconception catalogs
├── evaluation\
│ ├── metrics.py # Classification, Continuous, Retrieval metrics
│ ├── leakage_check.py # Zero data leakage assertion suite
│ ├── experiments.py # Experiments 1-9 runner
│ └── ablation.py # Component ablation runner
├── frontend\
│ └── app.py # Streamlit 4-page interactive UI
├── grading\
│ ├── llm_grader.py # Multi-signal evidence builder & structured LLM grader
│ ├── feedback.py # Formative feedback generator
│ └── scoring.py # Score normalization & threshold calibrations
├── nlp\
│ ├── preprocessing.py # Unicode NFKC, whitespace, sentence tokenization
│ ├── embeddings.py # Sentence-Transformers manager
│ ├── concept_extraction.py # spaCy noun chunking & soft semantic coverage
│ ├── misconception.py # Diagnostic misconception detector
│ └── language_detection.py # Multilingual detector (EN, HI, PA)
├── recommendation\
│ └── recommender.py # 4-part personalized learning path recommender
├── results\
│ ├── metrics\ # JSON metric summaries for all experiments
│ ├── tables\ # CSV experiment tables for publication
│ └── plots\ # Publication-ready PNG plots
├── tests\
│ └── test_pipeline.py # 10 automated unit & regression tests
├── requirements.txt # Package dependencies
├── .env.example # Environment variable template
├── README.md # Comprehensive documentation
└── run.py # Master CLI entrypoint📜 8. Citation & Academic Reference
@article{am_rag_asae_2026,
title={Adaptive Multilingual RAG-Based Explainable Automatic Short Answer Evaluation for Personalized Learning},
author={Prateek Dhar Dwivedi},
year={2026},
journal={arXiv preprint},
note={Experimental ASAG Framework with Multi-Signal Reranking and Misconception Diagnosis}
}