Modelpublic

mrzhao13/qwen3.5-2b-shopsimulator-grpo-rl500-1ep

sourceHugging Faceapache-2.0updated 1mo agoView on Hugging Face

0likes7downloads

Model Card

Qwen3.5-2B ShopSimulator GRPO RL-500 1 Epoch

This is the final Hugging Face checkpoint produced by one epoch of online GRPO on 500 ShopSimulator tasks. It starts from mrzhao13/qwen3.5-2b-shopsimulator-sft-512-1ep and uses Pi for multi-turn tool interaction through a modified Slime and ShopSimulator integration.

The post-training workload is text-only. The checkpoint retains the original Qwen3.5 multimodal architecture, but visual capabilities were not trained or evaluated.

Training

Item	Value
Original base model	`Qwen/Qwen3.5-2B`
RL initialization	`mrzhao13/qwen3.5-2b-shopsimulator-sft-512-1ep`
RL task set	`rl_500`, 500 unique tasks
Epochs	1
Candidates per task	4
Candidates / optimizer steps	2,000 / 100
Rollout batch size / global batch size	5 / 20
Learning rate	`1e-6`
Advantage estimator	GRPO
KL loss coefficient / type	`0.001` / `low_var_kl`
Maximum model turns	40
Maximum context / response length	16,384 / 2,048
Maximum tokens per GPU	12,288
Non-zero reward-variance groups	352 / 500 (70.4%)
Completed / truncated / aborted candidates	1,881 / 119 / 0
Training framework	Slime + Megatron-LM + SGLang + Pi
Hardware	One NVIDIA Pro 6000D 84 GB
Weight identity SHA-256	`8eff796683fa32bf74af7378a64646eea190e71cb52e397de431e7da72463f18`

Evaluation

All three checkpoints were evaluated with the same modified ShopSimulator service, deterministic price generation, fixed official_test_200 task slice, and one rollout per task. These are k=1 point estimates.

Model	Positive-reward pass@1	Strict-success pass@1	mean@1 `r_loose`	mean@1 `r_hard`
Qwen3.5-2B	2.0%	0.0%	0.004286	0.000000
SFT-512 1 epoch	72.5%	10.5%	0.389829	0.124417
This GRPO checkpoint	90.5%	31.0%	0.627786	0.354530

The RL checkpoint also achieved a 91.5% environment completion rate and a 40.0% right-product pass@1 rate on this fixed evaluation.

Usage

Use this repository as a drop-in replacement for Qwen/Qwen3.5-2B with a recent Transformers version that supports Qwen3.5. Refer to the official Qwen3.5-2B model card for loading and inference examples, replacing the model ID with this repository.

ShopSimulator evaluation requires the companion Pi extension, prompt template, patched environment and deterministic pricing behavior from the pi-slime-shopsimulator source repository when it is published.

Limitations

—Optimized only for Chinese ShopSimulator shopping tasks.
—Evaluated only in the patched text-based ShopSimulator environment.
—The deterministic pricing patch changes scores relative to the unpatched upstream environment.
—The official evaluation contains 200 tasks with one stochastic rollout each.
—No visual, broad instruction-following or safety evaluation was performed.
—Online RL can amplify environment- and reward-specific behavior.
—The SFT teacher data contains AI-generated content and may contain errors.

License and attribution

The original Qwen model, the SFT initialization and these derivative weights are distributed under Apache-2.0. The original Qwen license is included in LICENSE. This model card identifies the checkpoint as modified and does not grant rights to third-party ShopSimulator task or environment content.