ZHZisZZ/qwen3_5-4b.grpo.browser.use.i1.reasoning.wv16_overfit.from_sft

qwen35-4b.grpo.browser.use.i1.reasoning.wv16overfit.from_sft

GRPO from the browser.use.i1.reasoning SFT checkpoint on 16 WebVoyager read-only tasks, with train == eval by design. This was a memorization positive control for the RL pipeline, not a benchmark run. The interesting property found afterwards is that its behaviour change transfers to unseen tasks and unseen websites — see Measured results.

iter_9 is a rollout-9 snapshot, not the peak: SAVE_INTERVAL=10 meant the run's best eval (0.9297 at rollout 12) was never written to disk.

Base

.ckpts/pulled/sft.browser.use.i1.reasoning.webgym_gpt5_5_nogoto_wvclean@20260917-b-wvclean-eval/epoch_2

Reproduction prerequisites — READ FIRST

Two env vars in the command below do not exist in upstream `scripts/train/run_grpo.sh`. They were added by pod-local patches. On a clean checkout they are silently ignored:

(CUA_LITE_NORM_BY_TURNS, KL_LOSS_COEF are likewise patch-only; they are at their defaults here so they do not affect this run.) Either apply equivalent patches or edit run_grpo.sh directly before reproducing.

Also required:

# 1. a judge key — the env scores via a VLM judge at episode end; without it reset() fails loud
export OPENAI_API_KEY=...

# 2. an env-server with a warm 32-instance pool (SINGLETON backends are lazy by default)
WEBHARBOR_WEBVOYAGER_INSTANCES=32 uv run python scripts/serve_env.py \
  --port 30106 --env-ids webharbor.webvoyager --warm-singleton --token "$SESSION_ID"

Training command

Verbatim from the launcher. $W = repo root, $CKPT = the SFT base above, $DATA = the 16-task parquet, $P = browser.use.i1.reasoning, $CELL = grpo.$P.wv16_overfit.from_sft.

export CUA_LITE_ENV_SERVER_URL=http://127.0.0.1:30106
export CUA_LITE_ENV_SERVER_TOKEN=<session token>
export CUA_LITE_ALLOW_DIRTY_ENV_SERVER=1

CUDA_VISIBLE_DEVICES=0,1,2,3,4,5,6,7 NUM_TRAIN_GPUS=8 TP_SIZE=4 MBS=1 \
  MODEL_ID=Qwen/Qwen3.5-4B \
  HF_CKPT="$CKPT" \
  ENV_ID=webharbor.webvoyager \
  PROMPT_DATA="$DATA" \
  EVAL_PROMPT_DATA="$DATA" \
  ENV_CONCURRENCY=32 \
  ROLLOUT_BATCH_SIZE=16 \
  N_SAMPLES_PER_PROMPT=8 \
  NUM_STEPS_PER_ROLLOUT=8 \
  ROLLOUT_MAX_RESPONSE_LEN=2048 \
  ROLLOUT_TEMPERATURE=1.0 \
  LR=2e-6 \
  LR_WARMUP_ITERS=50 \
  CUA_LITE_MULTIMODAL_LAZY_EXPAND=1 \
  DISTRIBUTED_TIMEOUT_MINUTES=60 \
  EVAL_TEMPERATURE=1 \
  N_SAMPLES_PER_EVAL_PROMPT=8 \
  CONFIG_PATH="$W/devs/exps/train/browser/configs/qwen3_5/$P.yaml" \
  SAVE=1 NO_SAVE_OPTIM=1 SAVE_INTERVAL=10 EVAL_INTERVAL=1 NUM_ROLLOUT=40 \
  SAVE_HF_DIR="$W/.ckpts/qwen3_5-4b/$CELL/iter_{rollout_id}" \
  SAVE_DIR="/root/checkpoints/qwen3_5-4b/$CELL/megatron" \
  WANDB_GROUP_SUFFIX=".$CELL" \
  bash "$W/scripts/train/run_grpo.sh" < /dev/null

Derived: global_batch_size = 16 * 8 // 8 = 16 trajectories per optimizer step, 8 optimizer steps per rollout, train_iters = 40 * 16 * 8 // 16 = 320 total steps — so the 50-step warmup is 15.6% of the schedule. (Changing NUM_STEPS_PER_ROLLOUT changes train_iters; at nsp=1 it becomes 40 and Megatron's assert lr_warmup_steps < lr_decay_steps fails at init.)

Defaults in effect: --advantage-estimator grpo, --kl-loss-coef 0.00, --kl-coef 0.00, --entropy-coef 0.00, --lr-decay-style constant, --clip-grad 1.0, DROP_ZERO_STD_GROUP=1, CUA_LITE_NORM_BY_TURNS=0.

From the task config (browser.use.i1.reasoning.yaml): loop_detect: 5, enable_thinking: true, resolution 1280x720, extra_tools: ["back", "response"], image_max: 1. The 15-turn budget comes from the task metadata (max_steps, uniform across all 643 WebVoyager eval tasks), not from the config — it is central to the results below.

Runtime: 8xA100-80GB, TP=4 PP=1 DP=2. ~23 min per rollout (rollout generation dominates; optimizer steps are seconds), so iter_9 is ~3.5 h of training. Env errors during training ran 4-26 of 128 trajectories per rollout.

Training data

16 tasks drawn from the fixed 128-row read-only WebVoyager manifest with df.sample(n=16, random_state=42):

wolfram_alpha.36  amazon.10      bbc_news.8             booking.25
arxiv.27          apple.3        wolfram_alpha.6        arxiv.40
cambridge_dict.35 bbc_news.0     google_flights.33      wolfram_alpha.40
booking.2         booking.32     huggingface.6          google_search.12

(all prefixed webharbor.webvoyager@)

Identify this set by per-row `sha256(problem)`, not by the parquet's byte hash. Three different byte digests were observed for byte-identical row content across pods (c03d4cb7…, 11faf43b…, d7a4999e…) because a parquet embeds its writer's identity in the hashed bytes. The 16 rows were verified identical row-by-row across three independent builds.

Measured results

All scoring below was re-run with scripts/rollout.py at temperature 1.0 (the shipped config pins 0.0; greedy scoring is not comparable to a policy trained at 1.0 — at temperature 0 the n samples of a prompt are one trajectory re-run and pass rates collapse to the ends).

uv run python scripts/rollout.py \
  --model-id Qwen/Qwen3.5-4B --model-path <ckpt> \
  --env-id webharbor.webvoyager --prompt-data <parquet> \
  --group-size 4 --concurrency 32 \
  --config-path <browser.use.i1.reasoning.yaml with sampling_kwargs.temperature=1.0>

Metric definition. return_mean is valid-only: sum(valid_returns) / n_trajs_valid, i.e. trajectories that died in the env are excluded from the denominator rather than counted as 0. slime also logs a bare eval/<env>_eval which is the dense mean over n_expected (errored count as 0.0). The two coincide only when zero trajectories errored. Everything below is valid-only.

In-domain (the 16 trained tasks, 8 draws each)

Transfer (32 held-out tasks, 4 draws each, zero overlap with training)

Stratified by site, seed=42, round-robin across sites so no site dominates:

• —seen sites, unseen tasks (16): googlesearch.20/26, bbcnews.39/11, arxiv.28/14, huggingface.30/20, cambridgedictionary.31/37, googleflights.28/25, wolfram_alpha.25, booking.28, amazon.24, apple.20
• —unseen sites (16): allrecipes.39/14/32/34, espn.3/22/6, google_map.33/2/24, github.21/38/19, coursera.20/40/17

The SFT model was better on seen sites than unseen (0.650 vs 0.531); after GRPO the two are level (0.6875 vs 0.6833).

What changed

7 of the 16 training tasks were already 8/8 before any RL, so the in-domain gain could not exceed ~+0.35 however good the policy got; the observed +0.19 is roughly half of what was reachable. All of it came from the other 9 tasks.

The gain is almost entirely conversion of "ran out of turns without answering" into "answered". Every episode ends either because the agent emits a finish tool call (terminated=True) or because it exhausts the 15-turn budget (truncated=True):

Mean turns on success fell on 15 of 16 trained tasks, including the 7 that were already 8/8 and had no headroom. Episodes that terminate but answer wrongly rose from 13.7% to 21.0% — the behaviour is not free.

Group sizes n≈60 per transfer subgroup: the ALL-group pass gain and the truncation drop are robust (~4σ), but the seen/unseen difference is not statistically established (~1.3σ).

Caveats

• —train == eval during training. The training-internal eval curve (0.5417 → 0.7891 at rollout 9) is a memorization measurement and is not comparable to the numbers above, which come from a different scoring path; the absolute levels differ by ~0.1 and that discrepancy is unresolved.
• —The reward judge and the eval judge are the same VLM. A policy that learns to produce judge-pleasing final answers would be rewarded twice. The unseen-site transfer argues against pure judge-gaming, but the judge is unchanged there too.
• —Never evaluated on the 21 `mutating` tasks (3.3% of the 643-task eval split), which require real write actions. "Commit to an answer sooner" may be actively harmful there.
• —Offline mirror only (WebHarbor). No live-web evaluation.
• —generation_config.json is absent, inherited from the SFT export.
• —Sibling training-dynamics note (measured on the norm_by_turns=1 twin of this run): ~24% of optimizer steps produce exactly zero gradient — 38 of 160 over 20 rollouts, and flat (19/80 in each half) across a 4x improvement in env error rate. Zero-gradient steps still advance the LR schedule, so a meaningful slice of the 50-step warmup is spent on empty batches. This is a property of the trainer, not of this checkpoint, but it bounds how much of the nominal 320 steps actually moved the weights.