zhongweixie/showui-worldmodel-results

ShowUI + WorldModel: Training Results & Artifacts

This dataset contains evaluation results, world models, and training data from the ShowUI + WorldModel integration project.

📦 Contents

1. Evaluation Results (results/miniwob_predictions/)

• —Size: ~460MB
• —Format: JSONL files with episode-level predictions
• —Tasks: 9 MiniWoB++ tasks evaluated with ShowUI agent
• —Includes:
• —Task success/failure outcomes
• —Action predictions and execution traces
• —World model predictions (when enabled)
• —Screenshots and DOM snapshots

2. MiniWoB Metadata (datasets/miniwob_metadata/)

• —Size: ~134MB
• —Format: JSON files with task metadata
• —Files:
• —hf_train.json - Training set metadata (9,829 tasks)
• —hf_test.json - Test set metadata
• —Includes:
• —Task instructions and goals
• —Initial state descriptions
• —Screenshots (base64 encoded)
• —DOM structure

3. PatchWorld Trajectories (data/patchworld_trajectories/)

• —Size: ~996KB
• —Format: JSONL files in PatchWorld format
• —Tasks: 9 MiniWoB++ tasks
• —Includes:
• —State transitions with before/after observations
• —Action specifications
• —Success indicators
• —Split into train/test sets

4. Induced World Models (data/induced_world_models/)

• —Size: ~20KB
• —Format: Python modules
• —Models:
• —choose-date_world_model.py - Calendar date selection
• —grid-coordinate_world_model.py - Grid navigation
• —count-shape_world_model.py - Shape counting
• —Induced by: Claude Opus 3.7 via PatchWorld inducer

🎯 Use Cases

1. 1.Reproduce Evaluations: Use the prediction results to analyze ShowUI performance
2. 2.World Model Research: Study the induced world models for abstract task understanding
3. 3.Training Data: Use PatchWorld trajectories for imitation learning
4. 4.Metadata Analysis: Explore MiniWoB task structure and difficulty

📊 Key Findings

World Model Impact (Preliminary)

• —choose-date: World model correctly predicts calendar behavior
• —grid-coordinate: Spatial reasoning for grid navigation
• —count-shape: Accurate shape counting before action

Training Configurations

• —gc_fix: Gradient accumulation fix for stability
• —stratified: Balanced sampling across task difficulties

🔗 Related Resources

• —Code Repository: https://github.com/zhongweixie/guipatchworld
• —ShowUI Paper: arXiv:2411.17465
• —PatchWorld Framework: State transition modeling for GUI agents

📝 Citation

If you use this dataset, please cite:

@misc{showui-worldmodel-2024,
  title={ShowUI + WorldModel: Integrating World Models for GUI Agent Training},
  author={Xie, Zhongwei},
  year={2024},
  howpublished={HuggingFace Dataset: zhongweixie/showui-worldmodel-results}
}

@article{lin2024showui,
  title={ShowUI: One Vision-Language-Action Model for GUI Visual Agent},
  author={Lin, Kevin Qinghong and others},
  journal={arXiv preprint arXiv:2411.17465},
  year={2024}
}

🛠️ Technical Details

Evaluation Framework

• —Environment: MiniWoB++ (Chrome-based)
• —Agent: ShowUI-2B with LoRA adapters
• —World Model: PatchWorld-induced Python functions
• —Metrics: Task success rate, action accuracy

Data Processing

• —Trajectory conversion: convert_to_patchworld.py
• —World model induction: run_inducer.sh
• —Evaluation: eval_online.py with worldmodel_agent.py

Training Setup

• —Base model: ShowUI-2B
• —Training method: LoRA fine-tuning
• —Batch size: 1 per GPU (gradient accumulation: 8)
• —Learning rate: 1e-4
• —Epochs: 3

📧 Contact

For questions or issues, please open an issue on the GitHub repository.

License: Same as ShowUI (Apache 2.0) Last Updated: 2024-08-12