Modelpublic

ermiaazarkhalili/Qwen2.5-3B-GRPO-NuminaMath-100K

sourceHugging Faceapache-2.0updated 4mo agoView on Hugging Face

0likes

Model Card

Qwen2.5-3B-GRPO-NuminaMath-100K

This model is a fine-tuned version of ermiaazarkhalili/Qwen2.5-3B-SFT-UltraChat trained on the AI-MO/NuminaMath-CoT dataset using GRPO with LoRA adapters.

Overview

Qwen2.5-3B-GRPO-NuminaMath-100K is a language model optimized using GRPO. Group Relative Policy Optimization (GRPO) is an online RL method that optimizes policy using group-relative rewards without a critic model.

Key Features

—Reinforcement Learning: Trained with Group Relative Policy Optimization
—Mathematical Reasoning: Optimized on N/A math problems from NuminaMath-CoT
—Reward Function: Uses combined reward (accuracy + format)
—Efficient Training: Uses LoRA with 4-bit quantization
—Optimized for Inference: Available in GGUF quantizations for local deployment

Model Details

Property	Value
Developed by	ermiaazarkhalili
License	APACHE-2.0
Language	English
Base Model	ermiaazarkhalili/Qwen2.5-3B-SFT-UltraChat
Model Size	3B parameters
Tensor Type	BF16
Context Length	2,048 tokens
Training Method	GRPO with LoRA

Training Information

Training Configuration

Parameter	Value
Learning Rate	1e-06
Batch Size	2 per device
Gradient Accumulation Steps	8
Effective Batch Size	16
Number of Epochs	1
Max Sequence Length	2,048 tokens
LR Scheduler	Linear warmup + Cosine annealing
Warmup Ratio	0.1
Precision	BF16 mixed precision
Gradient Checkpointing	Enabled
Random Seed	42

LoRA Configuration

Parameter	Value
LoRA Rank (r)	16
LoRA Alpha	32
LoRA Dropout	0.05
Target Modules	qproj, kproj, vproj, oproj, gateproj, upproj, down_proj
Quantization	4-bit NF4

Training Metrics

Metric	Value
Hardware	NVIDIA H100 MIG

Dataset

This model was trained on the AI-MO/NuminaMath-CoT dataset.

Split	Samples
Training	N/A
Evaluation	N/A

Usage

Quick Start

python

from transformers import AutoModelForCausalLM, AutoTokenizer
import torch

model_id = "ermiaazarkhalili/Qwen2.5-3B-GRPO-NuminaMath-100K"

tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(
    model_id,
    torch_dtype=torch.bfloat16,
    device_map="auto"
)

messages = [
    {"role": "system", "content": "You are a helpful assistant."},
    {"role": "user", "content": "What is the sum of 2 + 2?"}
]

text = tokenizer.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
inputs = tokenizer(text, return_tensors="pt").to(model.device)

outputs = model.generate(**inputs, max_new_tokens=512, temperature=0.7, do_sample=True)
response = tokenizer.decode(outputs[0][inputs['input_ids'].shape[1]:], skip_special_tokens=True)
print(response)

Using Pipeline

python

from transformers import pipeline

generator = pipeline("text-generation", model="ermiaazarkhalili/Qwen2.5-3B-GRPO-NuminaMath-100K", device_map="auto")
messages = [{"role": "user", "content": "Explain the concept of machine learning."}]
output = generator(messages, max_new_tokens=256, return_full_text=False)
print(output[0]["generated_text"])

4-bit Quantized Inference

python

from transformers import AutoModelForCausalLM, BitsAndBytesConfig
import torch

quantization_config = BitsAndBytesConfig(
    load_in_4bit=True,
    bnb_4bit_compute_dtype=torch.bfloat16,
    bnb_4bit_use_double_quant=True,
    bnb_4bit_quant_type="nf4"
)

model = AutoModelForCausalLM.from_pretrained(
    "ermiaazarkhalili/Qwen2.5-3B-GRPO-NuminaMath-100K",
    quantization_config=quantization_config,
    device_map="auto"
)

GGUF Versions

For CPU or mixed CPU/GPU inference, GGUF quantized versions are available at: ermiaazarkhalili/Qwen2.5-3B-GRPO-NuminaMath-100K-GGUF

Using with Ollama

bash

ollama pull hf.co/ermiaazarkhalili/Qwen2.5-3B-GRPO-NuminaMath-100K-GGUF:Q4_K_M
ollama run hf.co/ermiaazarkhalili/Qwen2.5-3B-GRPO-NuminaMath-100K-GGUF:Q4_K_M "Hello!"

Limitations

—Language: Primarily trained on English data
—Knowledge Cutoff: Limited to base model's training data cutoff
—Hallucinations: May generate plausible-sounding but incorrect information
—Context Length: Fine-tuned with 2,048 token limit
—Safety: Not extensively safety-tuned; use with appropriate guardrails

Intended Use

Recommended Uses

—Research on language model fine-tuning
—Educational purposes
—Personal projects
—Prototyping conversational AI

Out-of-Scope Uses

—Production systems without additional safety measures
—Medical, legal, or financial advice
—Generating harmful or misleading content

Training Framework

—TRL: 0.24.0
—Transformers: 4.57.3
—PyTorch: 2.9.0
—Datasets: 4.3.0
—PEFT: 0.18.0
—BitsAndBytes: 0.49.0

Citation

bibtex

@misc{ermiaazarkhalili_qwen2.5_3b_grpo_numinamath_100k,
    author = {ermiaazarkhalili},
    title = {Qwen2.5-3B-GRPO-NuminaMath-100K: Fine-tuned Qwen2.5-3B-SFT-UltraChat on NuminaMath-CoT},
    year = {2026},
    publisher = {Hugging Face},
    howpublished = {\url{https://huggingface.co/ermiaazarkhalili/Qwen2.5-3B-GRPO-NuminaMath-100K}}
}

Acknowledgments

—Base model developers at ermiaazarkhalili
—Hugging Face TRL Team for the training library
—Dataset creators and contributors
—Compute Canada / DRAC for HPC resources

Contact

For questions or issues, please open an issue on the model repository.