tradexy/ml-hamz-colab1-grpo_unsloth
Uploaded model
- Developed by: tradexy
- License: apache-2.0
- Finetuned from model : unsloth/qwen2.5-3b-instruct-unsloth-bnb-4bit
This qwen2 model was trained 2x faster with Unsloth and Huggingface's TRL library.
ml-hamz-colab1-grpo_unsloth
Model Overview
This model is a fine-tuned version of the Qwen2.5-3B-Instruct architecture, optimized for instruction-following tasks. The fine-tuning process was conducted using the GRPO (Gradient Reversal Pretraining Optimization) method to enhance performance on specific queries.
Training Details
The fine-tuning was performed using the Unsloth Qwen2.5 (3B) GRPO Notebook-GRPO.ipynb). Key aspects of the training include:
- Dataset: Custom dataset focusing on instruction-based queries.
- Training Duration: Approximately 2 hours on a single GPU (T4 GPU - colab)
- Challenges: Managed limited disk space by deleting intermediate files during the process.
- Observation During Training: Reward increased as model answered questions correctly and completion_length became shorter.
Step - Training Loss - reward - rewardstd - completionlength - kl
1 - 0.000000 - 0.125000 - 0.000000 - 200.000000 - 0.000000
2 - 0.000000 - 0.072375 - 0.248112 - 200.000000 - 0.000000
3 - 0.000000 - -0.079000 - 0.163776 - 182.500000 - 0.000005
Usage Instructions
To utilize this model for inference:
from transformers import AutoTokenizer, AutoModelForCausalLM
tokenizer = AutoTokenizer.from_pretrained("tradexy/ml-hamz-colab1-grpo_unsloth")
model = AutoModelForCausalLM.from_pretrained("tradexy/ml-hamz-colab1-grpo_unsloth")
input_text = "Your instruction here"
inputs = tokenizer(input_text, return_tensors="pt")
outputs = model.generate(**inputs)
print(tokenizer.decode(outputs[0], skip_special_tokens=True))
For local deployment on Windows (can also use wsl ubuntu) see Machine Learning With Hamza
https://github.com/Hmzbo/Fine-tune-LLMS-with-grp