CoolFace
Modelpublic

budget-internalization-iclr2027/qwen3.5-4b-4k-grpo-budget-system-viableroughy-s250

sourceHugging Faceapache-2.0updated 3d agoView on Hugging Face
0likes15downloads
Model Card

Qwen3.5-4B · 4k-token budget · grpo-budget-system · viableroughy

RL-finetuned Qwen/Qwen3.5-4B trained for math reasoning under a 4,096-token generation budget that is stated to the model in a system prompt. Released as part of an anonymous ICLR 2027 submission.

Run ID (petname): `viableroughy` · checkpoint step 250

Note: this run was stopped at step 250 of the planned 300; other checkpoints in the budget sweep are at step 300.

Training

Base modelQwen/Qwen3.5-4B
AlgorithmGRPO (leave-one-out baseline, group reward normalization, token-level loss). Responses that hit the budget are truncated and scored as-is. The token budget is stated to the model in a system prompt.
Generation budget (max_new_tokens)4,096
DataDeepScaleR (math), 3 epochs max
Batch32 prompts × 8 rollouts per step
OptimizerAdam, cosine LR schedule, peak LR 5e-7, 10 warmup steps
Steps250
Rewardbinary answer correctness (\boxed{} extraction)
Weights dtypeBF16

Prompts are DeepScaleR problems with a statement of the 4,096-token budget placed in a system prompt, rendered with the base model's chat template.

Usage

python
from transformers import AutoModelForCausalLM, AutoTokenizer

repo = "budget-internalization-iclr2027/qwen3.5-4b-4k-grpo-budget-system-viableroughy-s250"
tok = AutoTokenizer.from_pretrained(repo)
model = AutoModelForCausalLM.from_pretrained(repo, torch_dtype="auto", device_map="auto")

With vLLM: vllm serve budget-internalization-iclr2027/qwen3.5-4b-4k-grpo-budget-system-viableroughy-s250

License

Inherits the license of the base model (Qwen/Qwen3.5-4B).