mobileforge-anonymous/mobileforge-benchmark-results

MobileForge Benchmark Results

Anonymous project: https://mobileforge-anonymous.github.io/ Anonymous code: https://github.com/mobileforge-anonymous/MobileForge

This dataset contains the evaluation artifacts used by MobileForge: Annotation-Free Adaptation for Mobile GUI Agents with Hierarchical Feedback-Guided Policy Optimization.

It includes AndroidWorld and MobileWorld GUI-only evaluation runs for the base agents and their MobileForge-adapted variants. The repository is intended for result verification, log inspection, and mapping the public model checkpoints to the exact benchmark artifacts reported in the paper.

<!-- mobileforge-release-inventory:start --> Release inventory archives: files=12; bytes=20015349616; sha256=b34dea1058bc00f8270e180e06b5d7707f6c9627f61e658dd738d2d8aad613de <!-- mobileforge-release-inventory:end -->

What is included?

• —archives/androidworld/*.tar.zst: AndroidWorld rollout artifacts for Qwen3-VL-8B, GUI-Owl-1.5-8B, ForgeQwen3-8B, and ForgeOwl-8B.
• —archives/mobileworld/*.tar.zst: MobileWorld GUI-only evaluation logs for the base and MobileForge-adapted models.
• —metadata/model_result_map.json: machine-readable mapping from public archive names to public checkpoints, artifact sizes, hashes, and headline results.
• —metadata/archives_manifest.json: compact manifest of all archive files and sizes.

The Hugging Face Dataset Viewer is disabled because the repository mainly stores compressed rollout logs rather than row-oriented examples.

AndroidWorld results

AndroidWorld uses 116 tasks and reports Pass@1, Pass@2, and Pass@3.

MobileWorld GUI-only results

MobileWorld GUI-only uses 117 tasks. No MobileWorld rollout, task, or feedback is used for MobileForge adaptation.

Directory layout

benchmark-results/
  metadata/
    archives_manifest.json
    model_result_map.json
  archives/
    androidworld/
      qwen3-vl-base.tar.zst
      qwen3-vl-mobileforge-200tasks.tar.zst
      qwen3-vl-mobileforge-400tasks.tar.zst
      qwen3-vl-mobileforge-900tasks.tar.zst
      gui-owl-base.tar.zst
      gui-owl-mobileforge-200tasks.tar.zst
      gui-owl-mobileforge-400tasks.tar.zst
      gui-owl-mobileforge-900tasks.tar.zst
    mobileworld/
      qwen3-vl-base.tar.zst
      qwen3-vl-mobileforge-900tasks.tar.zst
      gui-owl-base.tar.zst
      gui-owl-mobileforge-900tasks.tar.zst

Each archive expands to one experiment folder. AndroidWorld archives contain per-task compressed rollout artifacts (*.pkl.gz) and a run-level *_results.json summary. MobileWorld archives contain the corresponding MobileWorld evaluation logs and screenshots.

Download examples

Download only the metadata:

from huggingface_hub import hf_hub_download

path = hf_hub_download(
    repo_id="mobileforge-anonymous/mobileforge-benchmark-results",
    repo_type="dataset",
    filename="metadata/model_result_map.json",
)

Download one AndroidWorld run:

from huggingface_hub import snapshot_download

local_dir = snapshot_download(
    repo_id="mobileforge-anonymous/mobileforge-benchmark-results",
    repo_type="dataset",
    allow_patterns=[
        "archives/androidworld/qwen3-vl-mobileforge-900tasks.tar.zst",
        "metadata/model_result_map.json",
        "metadata/archives_manifest.json",
    ],
)

Extract an archive:

tar --use-compress-program=unzstd -xf archives/androidworld/qwen3-vl-mobileforge-900tasks.tar.zst

Notes

• —AndroidWorld summaries can be parsed with the AndroidWorld checkpoint parser released in the MobileForge codebase.
• —MobileWorld logs follow the upstream MobileWorld evaluation format.
• —Some source rollout pickle members were already truncated and unusable. In the anonymous archives, each such member is represented by a valid status placeholder; run-level result summaries are unchanged.
• —Compressed pickle files should be treated as executable Python serialization. Load them only in a trusted research environment.

Citation

Citation information is withheld during double-blind review.