karthik/verl-qwen2.5-0.5b-gsm8k-ppo-step360

VERL Fine-tuned Qwen2.5-0.5B on GSM8K (Step 360)

This model is a VERL (Volcano Engine Reinforcement Learning for LLMs) fine-tuned version of Qwen2.5-0.5B-Instruct on the GSM8K mathematical reasoning dataset using PPO.

Model Details

• —Base Model: Qwen/Qwen2.5-0.5B-Instruct
• —Training Method: VERL PPO
• —Dataset: GSM8K (Grade School Math 8K)
• —Training Steps: 360 steps
• —Framework: VERL

Training Progress

• —Initial Accuracy: ~1-2%
• —Step 20 Accuracy: 26.0%
• —Final Step 360: Latest checkpoint

Usage

from transformers import AutoTokenizer

# Note: This repository contains the tokenizer and config
# Model weights are in VERL/FSDP format and need conversion
tokenizer = AutoTokenizer.from_pretrained("karthik/verl-qwen2.5-0.5b-gsm8k-ppo-step360")

# For the full model, you would need to convert the VERL checkpoint:
# model = AutoModelForCausalLM.from_pretrained("Qwen/Qwen2.5-0.5B-Instruct")
# # Then load the VERL checkpoint weights

Training Details

This model was trained using the VERL framework with:

• —Algorithm: PPO (Proximal Policy Optimization)
• —Reward: GSM8K correctness scoring
• —Batch Size: 64-256 (adaptive)
• —Learning Rate: 1e-6 (actor), 1e-5 (critic)
• —Training Duration: 360 PPO steps
• —GPU Memory: 0.2-0.4 utilization

Performance

Shows significant improvement in mathematical reasoning:

• —Baseline: ~1-2% accuracy
• —After 20 steps: 26% accuracy
• —Continued training to 360 steps

Files Included

• —config.json - Model configuration
• —tokenizer.json - Tokenizer
• —tokenizer_config.json - Tokenizer configuration
• —vocab.json - Vocabulary
• —merges.txt - BPE merges
• —special_tokens_map.json - Special tokens
• —chat_template.jinja - Chat template

Model Weights

The actual model weights are stored in VERL/FSDP format and would need conversion for direct use with transformers. This repository provides the tokenizer and configuration for reference.

Citation

@misc{verl-qwen-gsm8k,
  title={VERL Fine-tuned Qwen2.5-0.5B on GSM8K},
  author={karthik},
  year={2024},
  howpublished={\url{https://huggingface.co/karthik/verl-qwen2.5-0.5b-gsm8k-ppo-step360}},
}

Framework

Trained using VERL - Versatile Reinforcement Learning framework.