HaoranLiu/SFT-4B-MobileGym

SFT-4B-MobileGym

Qwen/Qwen3-VL-4B-Instruct supervised-finetuned on GPT-5.5 teacher trajectories from `HaoranLiu/MobileGym`, trained with cua-lite + slime. Mobile GUI agent: 28 simulated Android apps, [0,1000]-normalized coordinates, 1080x2400 portrait viewport.

Results — MobileGym eval split

256 tasks (the whole split, no filter), 1 sample/task, greedy (temperature=0), max_steps: 45. MobileGym judges give a continuous progress rate in [0,1], so mean return and pass rate differ substantially.

\ Base figure supplied by the project owner and not re-measured here; whether it is a mean return or a pass rate was never confirmed, so compare it against 0.3656 or 26.2% accordingly. Both `iter_` rows were measured in this run under the protocol above.

The third epoch earns its keep

Per-task, iter_194 beats iter_129 on 37 tasks, loses on 12, ties on 207 (decoding is deterministic, so these are exact differences, not sampling scatter). The gain is broadest exactly where the model is weakest:

Note that training loss was nearly flat across epochs 2→3 (0.189 → 0.149) while downstream ability moved — the loss curve is not a useful stopping signal here.

Per-suite (this checkpoint)

reddit 1.000 · sms 1.000 · bilibili 0.611 · clock 0.611 · weather 0.611 · notes 0.600 · calendar 0.583 · spotify 0.515 · x 0.486 · filemanager 0.444 · wechat 0.439 · ebay 0.413 · account 0.400 · alipay 0.361 · railway12306 0.333 · redbook 0.333 · tencentmeeting 0.333 · wechatreading 0.295 · **crossappcommerce 0.274 · launcher 0.250 · map 0.196 · crossappwork 0.196** · **crossapplife 0.160 · crossapp_content 0.139**

Training

262 trajectories / 2124 steps, kept from MobileGym's 320 by episode_return > 0.6 and not exclude_reason (the exclude_reason clause matters on its own: 9 trajectories score 1.0 yet carry a footgun:loop / no_submit tag). 3 epochs, lr 5e-6 cosine, 4 trajectories/optimizer step, 195 steps, TP=2 on 2×H100 with optimizer CPU offload (required — it OOMs at step 1 without it). 91 minutes.

Rendered with scripts/configs/qwen3_vl/default/mobilegym.yaml — the same config used for evaluation, so training and serving see identical context. Use that config when running this model; the rendering (history protocol, chat template) is frozen into the training data.

Known weakness

The four crossapp_* suites are 87 of 256 eval tasks (34%) and the weakest, with only 4 solved outright. Single-app operation transfers well from this data; multi-app coordination largely does not — expected, since the training set is 262 mostly single-app trajectories averaging 8.1 steps. More cross-app teacher data is the next lever, not more epochs.

Full provenance — exact commands, data-quality audit, the memory/OOM analysis, and one export-code fix that HF-hosted datasets required — is in run_info.txt.

Usage

uv run python scripts/rollout.py \
  --model-id Qwen/Qwen3-VL-4B-Instruct \
  --model-path <path to this checkpoint> \
  --env-id mobilegym --splits eval \
  --config-path scripts/configs/qwen3_vl/default/mobilegym.yaml