HaoranLiu/DPO-4B-MobileGym

DPO-4B-MobileGym

Offline trajectory-level DPO on `HaoranLiu/SFT-4B-MobileGym` using `HaoranLiu/DPO-Qwen3-MobileGym`, trained with cua-lite + slime.

This checkpoint is `iter_137` — the end of epoch 2 of 3.

Results — MobileGym eval split

All 256 tasks (no filter), greedy (temperature=0), max_steps: 45, group_size=1. Every row was measured in this project under this one protocol.

The two metrics pick different checkpoints

MobileGym's judges emit a continuous progress rate in [0,1], so mean_return rewards partial progress while success_rate counts only fully solved tasks.

ep1 -> ep2   mean_return  0.3758 -> 0.3865   (+0.011)
             success         72   ->   71    (-1)
             partial         59   ->   65    (+6)

Epoch 2's mean gain comes entirely from six more tasks earning partial credit — it solves one fewer task outright than epoch 1, and epoch 3 changes neither count. Pick accordingly:

• —fully-completed tasks → iter_68 (epoch 1, and one third of the training cost)
• —average progress → this checkpoint
• —iter_206 has no argument in its favour

Data handling worth repeating

The dataset card flags ~18% exact-duplicate pairs (identical chosen and rejected action sequences, an artifact of pairing on source-row reference rather than content). Training on the raw file gives those pairs double gradient weight, so this run deduplicated by content hash first: 336 → 276 pairs.

Why epoch 2 still improves here (and does not everywhere)

The rejected trajectories in this dataset come from Qwen3-VL-4B-Instruct — the same model family being trained, i.e. on-policy negatives. In a companion experiment on lite.osworld, two runs from an identical SFT init differed only in the model that produced the rejected side:

They are indistinguishable after one epoch (0.3579 vs 0.3581) and diverge only in epoch 2. The training signal explains it: with 2B negatives the pairs separate much faster (epoch-2 median dpo_loss 4.37e-5 vs 1.48e-4; epoch-3 saturated steps 36.3% vs 28.4%), so later epochs spend their budget suppressing errors the 4B policy would never make.

Rule of thumb from these runs: on-policy negatives → 2 epochs; off-policy → 1.

Reproducibility

An earlier standalone 1-epoch run on the same data and parameters produced an iter_68 scoring 0.3793 / 72 successes — this run's epoch-1 checkpoint scored 0.3758 / 72, the same success count and mean within 0.0035. Run-to-run variance is small enough that the epoch-to-epoch differences above are not noise.

Full provenance in run_info.txt.

Usage

uv run python scripts/rollout.py \
  --model-id Qwen/Qwen3-VL-4B-Instruct \
  --model-path <path to this checkpoint> \
  --env-id mobilegym --splits eval \
  --config-path scripts/configs/qwen3_vl/default/mobilegym.yaml

The qwen3_vl adapter/config is required — that rendering is frozen into the training data.