pb09204048/Qwen3-4B-IF-RL-Specialist

Qwen3-4B IF RL Specialist

Standalone BF16 if specialist, trained from Qwen/Qwen3-4B with a rank-16 LoRA and merged into full Hugging Face weights.

These three specialists can be used as domain teachers for MOPD (Multi-Teacher On-Policy Distillation) experiments. The checkpoints were trained with domain RL; no MOPD student has been trained or evaluated in this release.

Training datasets and exposure

This repository's specialist uses [nvidia/Nemotron-Cascade-2-RL-data](https://huggingface.co/datasets/nvidia/Nemotron-Cascade-2-RL-data/tree/05bbaf03bac608e804efc86c5f8bf99844f2d197). Each specialist was trained on its own domain's data.

Preparation for this specialist

Only the IF-RL split is used for training. Preparation removes duplicate and evaluation-overlapping prompts, builds the supplied instruction checkers, and caps prompts at 4,096 tokens. After holding out 64 development prompts, 45,621 training prompts remain. The reward is the fraction of instruction constraints satisfied, using the supplied instruction IDs and checker arguments.

This checkpoint stopped before completing an epoch (77.86% of prepared candidates examined). This checkpoint sampled 35,520 prompts × 8 responses = 284,160 rollout responses. Sampled responses are not all gradient examples: constant-reward groups, zero centered advantages, and responses stopped at the length limit are excluded from updates. Counts refer to the training trajectory represented by the checkpoint and exclude evaluation and discarded/replayed recovery work.

All dataset revisions, prepared split sizes and hashes are in training/dataset_manifest.json. AIME24, the frozen lcbv5/test split, IFEval, and IFBench are evaluation datasets, not training splits. Each domain also has a separate 64-prompt development set. Dataset preparation removes normalized duplicates and evaluation overlaps; an additional exact prompt audit found no matches across the prepared training and evaluation sets. These checks do not establish semantic or pretraining decontamination.

Training method

• —Independent domain RL with GRPO-style group-centered advantages and a PPO clipped policy objective. Advantages subtract the prompt group's mean reward without standard-deviation normalization; there is no critic or KL penalty.
• —LoRA rank 16, alpha 32, dropout 0; attention projections, MLP projections, and lm_head adapted. Fresh LoRA/optimizer initialization for each specialist.
• —Up to 32 prompts × 8 responses per batch; rollout temperature 1.0, top-p 1.0, top-k disabled; thinking disabled; maximum response 16,384 tokens.
• —Adam learning rate 1e-5, 10-update warmup, clipping ratios [0.8, 1.2]. Responses receive equal weight, with their token losses averaged.
• —LoRA deltas were merged in FP32, then saved as BF16. Input embeddings were preserved and the adapted output head is untied. Export validation checked all 506 adapter tensors and exact logits after BF16 save/reload; see model_provenance.json.

Full cross-domain evaluation

All four models were evaluated on every benchmark under the same frozen protocol, for 8,308 responses total. Scores are percentages.

Math/coding: four sampled responses per prompt; temperature 0.7, top-p 0.8, top-k 20. Pass@1 is the mean success rate over those samples. IF: one greedy response per prompt; strict accuracy requires every instruction for that prompt to pass. All runs disable thinking and cap responses at 16,384 tokens, with a 24,576-token context.

The coding benchmark contains all 279 problems in the frozen DeepCoder `lcbv5/test` subset, not the entire current LiveCodeBench release. AIME24 includes all 30 problems; IFEval includes all 541 prompts; IFBench includes all 300 prompts. Coding accuracy requires all tests to pass (4-second wall, 3-second CPU, 1-GiB memory limit per test). Two inherited IFEval letter-frequency checks randomly substituted letters for literal # and !; the final scorer counts the requested punctuation uniformly for all models and retains every prompt.

These are latest checkpoints, not checkpoints selected to maximize the reported test scores. Code and IF have incomplete and unequal training exposure. Math and coding gains come with longer responses: their domain specialists use 2.09× and 3.87× the base model's mean response tokens. Paired 95% intervals exclude zero for the math/AIME24, code/coding, and IF/IFBench gains; the IF specialist's IFEval gain remains uncertain (−0.18 to +6.47 percentage points). Intervals are exploratory, without adjustment for multiple tests.

• —Full report, pass@4, response lengths and limitations
• —Machine-readable metrics · Comparison CSV · Paired confidence intervals
• —Frozen evaluation protocol · IFEval checker correction

Use as MOPD teachers

Use a student initialized from the compatible Qwen3-4B base and route each prompt to the teacher for its known domain. Generate continuations with the student, then score those student-generated tokens under the corresponding teacher to obtain token-level supervision for the chosen on-policy distillation objective. Keep tokenizer IDs, chat templates, thinking mode, and token masks aligned. These checkpoints provide the teacher models; this release does not prescribe a complete MOPD optimizer or claim a distilled-student result. See the MOPD paper for the multi-teacher on-policy formulation.

• —Math teacher
• —Code teacher
• —Instruction-following teacher

Repository IDs and compatible settings are available in mopd_teachers.json. The domain comparison above supplies teacher baselines for capability-integration experiments.

Inference

The exports were validated with Transformers 5.12.1. Use a compatible Transformers release with PyTorch and Accelerate.

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

repo_id = "pb09204048/Qwen3-4B-IF-RL-Specialist"
tokenizer = AutoTokenizer.from_pretrained(repo_id)
model = AutoModelForCausalLM.from_pretrained(
    repo_id, dtype=torch.bfloat16, device_map="auto"
)
inputs = tokenizer.apply_chat_template(
    [{"role": "user", "content": "Reply with exactly two words: hello world"}],
    tokenize=True, add_generation_prompt=True, enable_thinking=False,
    return_dict=True, return_tensors="pt",
).to(model.device)
with torch.inference_mode():
    output = model.generate(**inputs, max_new_tokens=16384, do_sample=False)
print(tokenizer.decode(output[0, inputs.input_ids.shape[-1]:], skip_special_tokens=True))

Disabling the thinking template does not prevent ordinary step-by-step reasoning in response text. Benchmark numbers apply to the specific prompts and grading protocol above.

Attribution and license

The base model is Qwen/Qwen3-4B, revision 1cfa9a7208912126459214e8b04321603b3df60c. Its Apache-2.0 license is retained in LICENSE. This release attributes DAPO-Math-17k (the linked snapshot has no license field in its Hub metadata), DeepCoder-Preview-Dataset (MIT), and Nemotron-Cascade-2-RL-data (ODC-BY). Dataset terms remain those of the linked source repositories; training corpora are not redistributed here.