Modelpublic

amayuelas/Qwen3.5-4B-MatRL-MT-RL

sourceHugging Faceapache-2.0updated 23d agoView on Hugging Face

0likes256downloads

Model Card

Qwen3.5-4B — MatRL multi-turn RL

Multi-turn RL on top of `amayuelas/Qwen3.5-4B-MatRL-MT-SFT`, for crystal-structure inverse design. The agent proposes candidate structures, evaluates them against a machine-learning interatomic potential, refines, and submits — rewarded on validity, novelty, and thermodynamic stability.

Results

SUN rate (Stable, Unique, Novel) — n=100 prompts × r=8 = 800 rollouts/track. equflashv2 is the reward-path judge and therefore the reference; orb-mpa and eqv3 are held-out judges that never touched training. Judging protocol is independent of the reward path (200 relaxation steps, fmax 0.02).

track	equflashv2 (reference)	orb-mpa	eqv3	held-out consensus
`mt_id` (in-distribution)	15.5%	16.8%	17.8%	14.4%
`mt_ood` (out-of-distribution)	15.1%	17.4%	17.9%	15.2%
`mt_uncond` (unconditional)	17.0%	20.1%	24.6%	19.8%

The base model scores 0% on these tracks under strict scoring — it almost never calls submit (3 times in 2,880 rollouts), so nothing is reconstructed to score. Learning to commit within the turn budget is most of the gap.

On reward hacking

The policy is optimized against equflashv2, so the number that matters is whether judges it never saw agree:

track	reward judge	held-out consensus	gap
`mt_id`	15.5%	14.4%	+1.1 pp
`mt_ood`	15.1%	15.2%	−0.1 pp
`mt_uncond`	17.0%	19.8%	−2.8 pp

Held-out judges track the reward judge closely and exceed it on the unconditional track. The gains are not artifacts of the training judge.

Training


init	`amayuelas/Qwen3.5-4B-MatRL-MT-SFT`
algorithm	GRPO-style multi-turn RL, 10 turns / 8 evaluations per episode
reward	validity + novelty + relaxed stability (equflashv2 MLIP, 20 steps / fmax 0.1)
steps	200
seq len	16,384
precision	bf16
trainer	prime-rl

Trained on 8×A100-40GB.

Known training caveat

Steps ~150–200 ran with 9–15% of each batch clipped at the trainer's 16,384-token window, so those episodes' endings — including the submit turn — contributed no gradient. This checkpoint is released as-is because the held-out evaluation above shows it is not inflated: independent judges match or exceed the reward judge on every track. A truncation-clean redo of the final 50 steps was trained separately and scored lower (13.5 / 12.5 / 10.8%), so the clipping does not appear to have produced a falsely strong model.

Important: thinking channel

Reasoning lives in the native thinking channel, preserved across tool calls. Do not sample with enable_thinking=false — it disables the behavior this model was trained to use.

Serving

Includes preprocessor_config.json / video_preprocessor_config.json — vLLM requires them for this VL-capable model class even for text-only serving.