Flynntaggart26/qwen2.5-scholarship-rag-adapter

Qwen2.5-0.5B Scholarship & Higher Education RAG Adapter 🎯

   

Executive Summary & Model Description

The Qwen2.5-0.5B Scholarship & Higher Education RAG Adapter is a specialized, Parameter-Efficient Fine-Tuned (PEFT) Low-Rank Adaptation (LoRA) module engineered to enhance Retrieval-Augmented Generation (RAG) pipelines in domain-specific financial aid, tuition net price analysis, and higher education ROI evaluation.

Standard general-purpose Large Language Models (LLMs) frequently struggle with numeric extraction, context hallucination, and temporal drift when processing complex institutional disclosures (e.g., U.S. Department of Education College Scorecard and IPEDS data). By aligning the Qwen2.5-0.5B-Instruct base architecture with domain-curated datasets, this model achieves high structural faithfulness, precise entity extraction, and strict context adherence.

Architectural Overview & RAG Integration

This adapter acts as the domain-specific generation head within a modular RAG framework: [ User Query ] │ ▼ [ Vector Retriever / Hybrid Search ] ──► (Fetches top-k chunks from College ROI Dataset) │ ▼ [ Context Ingestion & Prompt Formatting ] │ ▼ [ Base Model: Qwen2.5-0.5B + LoRA Adapter ] ◄── (Loaded via PEFT) │ ▼ [ Structured, Faithfully Grounded Response ]

Technical Specifications & Mathematical Formulation

• —Developer: Egehan Şahban
• —Model Architecture: Causal Language Model Adapter via Low-Rank Adaptation (LoRA)
• —Base Architecture: Qwen/Qwen2.5-0.5B-Instruct
• —Fine-Tuning Paradigm: PEFT / QLoRA (4-bit quantization base with FP16/BF16 adapter weights)
• —Primary Domain: U.S. Higher Education Financial Aid, Institutional Metrics, and Decision Support Systems
• —Language: English

LoRA Parameter Update Formula

During fine-tuning, the base model frozen weight matrix $W_0 \in \mathbb{R}^{d \times k}$ is augmented by a low-rank decomposition matrix product $B \cdot A$:

$$h = W0 x + \Delta W x = W0 x + \frac{\alpha}{r} B A x$$

where $r = 16$ represents the rank and $\alpha = 32$ represents the scaling factor.

Training Methodology & Hyperparameters

The adapter was fine-tuned using Hugging Face transformers, peft, and bitsandbytes libraries.

LoRA Configuration

• —Rank ($r$): 16
• —Alpha ($\alpha$): 32
• —Dropout Rate: 0.05
• —Target Projection Modules: q_proj, k_proj, v_proj, o_proj, gate_proj, up_proj, down_proj
• —Bias: none

Fine-Tuning Execution Hyperparameters

Quickstart & Code Example

You can easily instantiate and run inference with this adapter using Hugging Face transformers and peft:

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer
from peft import PeftModel

base_model_id = "Qwen/Qwen2.5-0.5B-Instruct"
adapter_id = "Flynntaggart26/qwen2.5-scholarship-rag-adapter"

# 1. Load Base Model and Tokenizer
tokenizer = AutoTokenizer.from_pretrained(base_model_id, trust_remote_code=True)
base_model = AutoModelForCausalLM.from_pretrained(
    base_model_id,
    torch_dtype=torch.float16,
    device_map="auto",
    trust_remote_code=True
)

# 2. Integrate LoRA Adapter
model = PeftModel.from_pretrained(base_model, adapter_id)
model.eval()

# 3. Construct Context-Grounded RAG Prompt
prompt = """<|im_start|>system
You are an expert higher education financial advisor. Answer the query strictly based on the provided context.<|im_end|>
<|im_start|>user
Context:
MIT provides Need-Blind admissions for all undergraduate applicants regardless of citizenship. Stanford University offers Need-Blind admissions for international students and covers 100% of demonstrated financial need.

Query:
Compare the international financial aid policies of MIT and Stanford.
<|im_end|>
<|im_start|>assistant
"""

# 4. Generate Output
inputs = tokenizer(prompt, return_tensors="pt").to(model.device)
with torch.no_grad():
    outputs = model.generate(
        **inputs,
        max_new_tokens=150,
        temperature=0.1,
        top_p=0.9,
        repetition_penalty=1.1
    )

response = tokenizer.decode(outputs[0][inputs.input_ids.shape[1]:], skip_special_tokens=True)
print(response)

Benchmarks & Performance MetricsEvaluation was conducted against the base model using automated RAG evaluation metrics (Faithfulness, Numeric Extraction Accuracy, and Hallucination Rate) over the learn-rag-eval-qa dataset:Evaluation MetricBase Model (Qwen2.5-0.5B)Base + Adapter (This Model)Relative DeltaContext Faithfulness78.4%94.2%+15.8%Numeric Accuracy81.0%96.5%+15.5%Hallucination Rate14.2%2.8%-11.4%Context Recall75.1%91.8%+16.7%

Citation & Contact Information
If you utilize this model adapter, training methodology, or associated research datasets in your work, please cite it as follows:
@misc{sahban2026qwen25rag,
  author = {Şahban, Egehan},
  title = {Qwen2.5 Scholarship & Higher Education RAG Adapter},
  year = {2026},
  publisher = {Hugging Face},
  journal = {Hugging Face Repository},
  howpublished = {\url{[https://huggingface.co/Flynntaggart26/qwen2.5-scholarship-rag-adapter](https://huggingface.co/Flynntaggart26/qwen2.5-scholarship-rag-adapter)}}
}

Author: Egehan Şahban

GitHub: Flynntaggart26

Hugging Face: Flynntaggart26

LinkedIn: Egehan Şahban