Modelpublic

vishnuamarapu/Full-Fine-Tuning-Qwen-2.5-0.5B-instruct-QLora

sourceHugging Faceapache-2.0updated 3mo agoView on Hugging Face

0likes

Model Card

QLoRA Fine-Tuned Qwen2.5-0.5B-Instruct

This repository contains a QLoRA (Quantized Low-Rank Adaptation) fine-tuned version of Qwen2.5-0.5B-Instruct.

The model was trained using QLoRA, where the base model was loaded in 4-bit precision during training to reduce GPU memory usage. Only the LoRA adapter weights were updated and are included in this repository.

Note: This repository contains only the QLoRA adapter. During inference, the adapter can be loaded with the original base model in FP32, FP16, BF16, 8-bit, or 4-bit depending on your hardware.

Model Details

—Base Model: Qwen/Qwen2.5-0.5B-Instruct
—Fine-Tuning Method: QLoRA (PEFT)
—Framework: Hugging Face Transformers + PEFT
—Task: Conversational Text Generation

Installation

bash

pip install transformers peft accelerate torch safetensors

Loading the Model

python

import torch
from transformers import AutoTokenizer, AutoModelForCausalLM
from peft import PeftModel

BASE_MODEL = "Qwen/Qwen2.5-0.5B-Instruct"
ADAPTER = "vishnuamarapu/Full-Fine-Tuning-Qwen-2.5-0.5B-instruct-QLora"

tokenizer = AutoTokenizer.from_pretrained(BASE_MODEL)

base_model = AutoModelForCausalLM.from_pretrained(
    BASE_MODEL,
    torch_dtype=torch.float32
)

model = PeftModel.from_pretrained(
    base_model,
    ADAPTER
)

model.eval()

Example Inference

python

messages = [
    {
        "role": "system",
        "content": (
            "You are Vishnu's personal AI assistant. "
            "Answer questions about Vishnu."
        )
    },
    {
        "role": "user",
        "content": "Tell me about Vishnu."
    }
]

text = tokenizer.apply_chat_template(
    messages,
    tokenize=False,
    add_generation_prompt=True
)

inputs = tokenizer(
    text,
    return_tensors="pt"
)

with torch.no_grad():

    outputs = model.generate(
        **inputs,
        max_new_tokens=256,
        temperature=0.7,
        do_sample=True,
        top_p=0.9,
    )

response = tokenizer.decode(
    outputs[0][inputs.input_ids.shape[-1]:],
    skip_special_tokens=True,
)

print(response)

Gradio Demo

python

import torch
import gradio as gr

from transformers import AutoTokenizer, AutoModelForCausalLM
from peft import PeftModel

BASE_MODEL = "Qwen/Qwen2.5-0.5B-Instruct"
ADAPTER = "vishnuamarapu/Full-Fine-Tuning-Qwen-2.5-0.5B-instruct-QLora"

tokenizer = AutoTokenizer.from_pretrained(BASE_MODEL)

base_model = AutoModelForCausalLM.from_pretrained(
    BASE_MODEL,
    torch_dtype=torch.float32
)

model = PeftModel.from_pretrained(
    base_model,
    ADAPTER
)

model.eval()

SYSTEM_PROMPT = (
    "You are Vishnu's personal AI assistant. "
    "Answer questions about Vishnu."
)

def predict(message, history):

    messages = [
        {
            "role": "system",
            "content": SYSTEM_PROMPT,
        }
    ]

    for user, assistant in history:
        messages.append({"role": "user", "content": user})
        messages.append({"role": "assistant", "content": assistant})

    messages.append({"role": "user", "content": message})

    text = tokenizer.apply_chat_template(
        messages,
        tokenize=False,
        add_generation_prompt=True,
    )

    inputs = tokenizer(
        text,
        return_tensors="pt",
    )

    with torch.no_grad():
        outputs = model.generate(
            **inputs,
            max_new_tokens=256,
            temperature=0.7,
            do_sample=True,
            top_p=0.9,
        )

    response = tokenizer.decode(
        outputs[0][inputs.input_ids.shape[-1]:],
        skip_special_tokens=True,
    )

    return response

gr.ChatInterface(
    fn=predict,
    title="Vishnu Personal AI (QLoRA)",
    description="Qwen2.5-0.5B fine-tuned using QLoRA",
).launch()

Generation Parameters

Parameter	Value
maxnewtokens	256
temperature	0.7
top_p	0.9
torch_dtype	float32 (CPU Example)

Repository Structure

adapter_config.json
adapter_model.safetensors
chat_template.jinja
tokenizer.json
tokenizer_config.json
README.md

Training Overview

This model was fine-tuned using QLoRA (Quantized Low-Rank Adaptation).

During training:

—The base model was loaded in 4-bit NF4 quantization.
—Only the LoRA adapter weights were trained.
—The original model weights remained frozen.
—QLoRA significantly reduced GPU memory requirements while maintaining strong model performance.

The uploaded repository contains only the adapter weights. During inference, these adapters are applied to the original Qwen2.5-0.5B-Instruct base model.

Citation

If you use this model in your work, please cite this repository.

bibtex

@misc{vishnu_qwen25_qlora,
  author = {Vishnu Amarapu},
  title = {QLoRA Fine-Tuned Qwen2.5-0.5B-Instruct},
  year = {2026},
  publisher = {Hugging Face},
  howpublished = {\url{https://huggingface.co/vishnuamarapu/Full-Fine-Tuning-Qwen-2.5-0.5B-instruct-QLora}}
}

Acknowledgements

—Alibaba Cloud Qwen Team for the base model.
—Hugging Face Transformers.
—PEFT (Parameter-Efficient Fine-Tuning).
—BitsAndBytes for efficient 4-bit quantized training.
—PyTorch.