AnjanSB/Qwen2.5-3B-Instruct-NQ-RAG-DPO-LoRA

Introduction

This repository contains a LoRA adapter fine-tuned using Direct Preference Optimization (DPO) over a Retrieval-Augmented Generation (RAG) evaluation pipeline built on the Natural Questions validation set.

Training Pipeline

1. 1.Base Model: Qwen/Qwen2.5-3B-Instruct

1. 1.RAG responses generated over NQ validation split

1. 1.Responses scored using custom reward signals:

• —Faithfulness

• —Citation usage

• —Hallucination detection

• —Refusal detection

1. 1.Preference pairs constructed using margin filtering

1. 1.LoRA fine-tuning using DPO

Dataset Lineage

This model is trained and evaluated using:

Dataset repository:

AnjanSB/NQ-RAG-DPO-Evaluation

Configurations used:

• —rag_responses (base + trained generations)

• —responses_scores (reward signals)

• —dpo_train_data (preference dataset)

• —comparison_metrics (evaluation results)

Evaluation Summary

The Train Data which is used to train base model indicates small in size(228 Total records) and high quality(2.18 a significant jump of mean response margin between choosen and rejected responses of a same prompt).

The trained model shows:

• —Significantly improved responses margin which indicates stable responses.

• —Improved citation usage

• —Reduced hallucination

• —Slightly improved faithfulness

• —Stable refusal behavior

👤 Author

AnjanSB

Experiment Repo : https://dagshub.com/AnjanSB/RAG-DPO-PEFT-LLMOPS

Profile : https://www.linkedin.com/in/anjansb/