RLMM/zerohero-craftax-qwen3-13m-strong-ppo-horizon30

ZeroHero Craftax strong PPO: horizon 30, cycle 300

This is the final cycle-300 checkpoint of a 12,969,600-parameter Qwen3-architecture executor for Craftax-Dialogue hidden-goal navigation. The small model was trained from random initialization, then supervised on expert trajectories; online training starts from strong SFT checkpoint 400. It does not inherit pretrained Qwen3 weights. The final checkpoint was fixed in advance and was not selected by held-out success rate.

Task and horizon

Every response consumes one of 30 decisions, including questions, malformed actions and blocked moves. Training, controller, feedback, final evaluation and the matched expert all use this horizon. The original 75-decision releases are a different protocol and remain separate.

The executor sees a local symbolic map and dialogue, emits reasoning and an action, and can ask a question. Only the private Gemma question oracle sees the hidden navigation target. Gemma also supplies public-state action labels for DAgger. The response format and reasoning are retained without output repair. This is a specialized policy, not a general-purpose chatbot or standalone oracle.

Training and architecture

• —Method: PPO; seed 93001; 300 cycles from the original strong SFT.
• —Four-cycle operational pilots use separate outputs; long training starts fresh from SFT and does not inherit pilot weights. No success-rate selection gate.
• —Recorded training episodes: 38400; decisions: 998845.
• —Six layers, hidden size 384, intermediate size 1264; six attention heads and three key/value heads; 2,048-token context and custom 4,096-token tokenizer.
• —Released model weights are FP32. Scalar training settings are in training_config.json; source and file hashes are in provenance.json.
• —PPO uses 128 episodes and two optimizer steps per cycle. DAgger uses 64 episodes, 512 expert labels and 2,048 supervision samples with 32 steps. BATS uses LinUCB to select between these same two update recipes.
• —Learner source commit: 7ca74cf7ae851e3bfd0e34885e6d63df2a87b3a9.

Environment, prompt formatter and action parser are provided by ZeroHero. Full interactive evaluation requires those components and the Gemma oracle.

Matched held-out evaluation

Mean episode length: 24.6113 decisions. All 512 unique world seeds and hidden targets match the expert evaluation. Students sample at temperature 1; the expert uses temperature 0. Both use a private Gemma question oracle, whose answers need not be bitwise identical between runs. No target repairs or technical step errors occurred. These are single-training-seed policy-plus-oracle results, not standalone question-answer accuracy or standard Craftax achievement scores. See evaluation.json for confidence intervals and exploratory paired comparisons.

Load and generate one response

Use Transformers with Qwen3 support (verified with Transformers 4.51.1). Use the raw prompt format, not a generic chat template.

from pathlib import Path
from huggingface_hub import hf_hub_download
from transformers import AutoModelForCausalLM, AutoTokenizer

repo_id = "RLMM/zerohero-craftax-qwen3-13m-strong-ppo-horizon30"
tokenizer = AutoTokenizer.from_pretrained(repo_id)
model = AutoModelForCausalLM.from_pretrained(repo_id, torch_dtype="auto").eval()
prompt = Path(hf_hub_download(repo_id, "example_prompt.txt")).read_text()
inputs = tokenizer(prompt, return_tensors="pt", return_token_type_ids=False)
output = model.generate(
    **inputs, do_sample=True, temperature=1.0, top_k=0, top_p=1.0,
    max_new_tokens=256, pad_token_id=tokenizer.pad_token_id,
    eos_token_id=tokenizer.eos_token_id,
)
print(tokenizer.decode(output[0, inputs["input_ids"].shape[1]:], skip_special_tokens=True))

value_head.pt contains the separate training value head; AutoModelForCausalLM loads only the language policy. This inference release does not contain optimizer, RNG, replay or bandit recovery state. sha256.json lists packaged-file hashes.