witcheer/qwen3-4b-gsm8k-grpo

Qwen3-4B GSM8K GRPO adapter (WITCHEER)

A QLoRA adapter for unsloth/Qwen3-4B, trained with GRPO (Group Relative Policy Optimization) on GSM8K. Reinforcement learning, not supervised fine-tuning: the model practices against a reward instead of copying answers.

Result

Mean training reward climbed 1.375 -> 1.725 over 150 steps (num_generations=8), still rising at the end (not saturated). Delta measured with the same in-process eval (greedy, identical prompt + answer-extraction) on base and tuned — the delta is the claim, not the absolute %.

Training

• —Method: GRPO via TRL, run on plain HF generation (use_vllm=False).
• —Why no vLLM: the FP8 / vLLM fast-RL path is blocked on a CUDA-toolkit-less RTX 5090 (Blackwell) three ways — flashinfer FP8 kernels need CUDA >= 12.9 (box has 12.8); the only Blackwell-wheel vLLM forces an unsloth that crashes at LoRA warmup; and an irreconcilable torch pin (unsloth 2026.6.1 -> torch 2.10 vs vLLM 0.21 -> torch 2.11+cu130). The use_vllm=False path sidesteps all three and runs on the box as-is.
• —Policy: Qwen3-4B 4-bit QLoRA (r=32, alpha=32, all attn+MLP projections).
• —Reward: correctness (GSM8K answer match) + format bonus (ends in #### <answer>).
• —Prompt: identical to the eval prompt (train/eval alignment).
• —Steps: 150 · num_generations 8 · Hardware: single RTX 5090 32GB · Unsloth 2026.6.1, torch 2.10+cu128.

Usage

from unsloth import FastLanguageModel
model, tok = FastLanguageModel.from_pretrained("unsloth/Qwen3-4B", load_in_4bit=True)
model.load_adapter("witcheer/qwen3-4b-gsm8k-grpo")

Scope

In-process eval, internally consistent (base vs tuned, identical harness) — not the leaderboard server harness, so the absolute % is not directly comparable across setups. A few points of measured GSM8K from RL on a 4B; the reward sharpened existing capability rather than adding a new tier.

Rig: https://github.com/notwitcheer/llm-bench-rig