amayuelas/Qwen3.8-27B-MatRL-MT-RL

Qwen3.8-27B-MatRL-MT-RL

Multi-turn RL for novel inorganic crystal structure generation, trained with tool use against a machine-learned interatomic potential (MLIP) reward.

This is the 27B arm of the MatRL scale series. The 4B and 9B arms are Qwen3.5-4B-MatRL-MT-RL and Qwen3.5-9B-MatRL-MT-RL.

Intermediate checkpoint. This is step 100 of RL stage 2c — roughly 170 of a planned 400 cumulative RL steps. It is released as a work-in-progress snapshot, not a final model. See Known issues before using it.

Base model

Built on `Qwen/Qwen3.8-27B` (27.36B params, 64 layers, hidden size 5120, hybrid linear/full attention).

Note that config.json reports model_type: "qwen3_5" and Qwen3_5ForConditionalGeneration — Qwen3.8 is a version bump of the Qwen3.5 architecture, not a new one. The tokenizer and chat template are Qwen3.8's and differ from Qwen3.5's; use the ones shipped in this repo rather than substituting Qwen3.5's.

Training pipeline

Cumulative: 170 RL steps on top of the SFT.

Task and reward

The model works in a multi-turn tool loop (up to 10 turns) proposing crystal structures, then submits a final candidate. Task mix: unconditional generation, target formula, target element set, and property-conditioned targets (band gap, bulk modulus).

Tools: retrieve_materials, propose_structure, substitute_element, check_novelty, evaluate_structure, predict_band_gap, calculate_bulk_modulus, submit.

Reward is a weighted rubric over submission, validity, novelty, honest stability (MLIP relaxation, equflashv2 backend), SUN bonuses, property-target match, group diversity, efficiency, and format.

Training setup

• —Framework: prime-rl 0.7.0, GRPO-style with group-relative advantages
• —Hardware: 8×H200 (5 trainer / 2 inference / 1 MLIP reward pool)
• —Batch: 128 rollouts, group size 16, oversampling 1.25
• —Sampling: temperature 1.15, max_completion_tokens 2048, max_model_len 32768
• —Optimizer: fp32 optimizer state, constant LR, FSDP with CPU offload

Known issues

• —Intermediate, not converged. Mean reward is roughly flat across stage 2c (~3.2 → ~3.4), within run-to-run noise.
• —Tool-call batching drift. Over stage 2c the policy progressively packs more tool calls into a single assistant message (~3 at step 1 → ~7 by step 100, with batches up to 27). Around 41% of rollouts at this checkpoint hit the 2048-token completion cap mid-batch, which can emit a truncated final tool call. Downstream users should expect occasional malformed trailing tool calls and may want a larger completion budget or a cap on tool calls per message.
• —The stage-2c run later terminated at step 122 (vLLM engine timeout on very long prompts, a downstream consequence of the same drift). This checkpoint predates that failure.

Usage

from transformers import AutoModelForCausalLM, AutoTokenizer

model_id = "amayuelas/Qwen3.8-27B-MatRL-MT-RL"
tok = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(model_id, dtype="auto", device_map="auto")

The model expects the multi-turn tool-calling format it was trained in; use the bundled chat_template.jinja.

License

Inherits the license of the base model, Qwen/Qwen3.8-27B.