figai/GUI-Perturbed-Finetuned-Result-Viewer

GUI-Perturbed: Fine-tuned Results Viewer

We fine-tuned UI-TARS-1.5-7B on the GUI-Perturbed dataset using low-rank adaptation (LoRA). The aim was to test whether training on the perturbations directly recovers the accuracy the baseline models lose under them.

We ran this across several configurations:

• —Augmentation type: style, text-shrink, precision, and combined.
• —Data scale: 6.5k and 25k training samples.
• —Data source: synthetic perturbations (GUI-Perturbed) and a real, diverse mix (the Salesforce GUI-grounding dataset).

Each configuration was evaluated on both GUI-Perturbed and ScreenSpot-v2. In total the viewer holds 6,240 results per model. Our analysis on the results can be found in the technical write-up here.

How to read it

Filter by augmentation type, data scale, data source, outcome, and whether reasoning was applied. Each result shows the perturbed webpage, the instruction, the fine-tuned model's predicted click against the baseline, and whether it landed on the correct element.

The rest of the work

• —Technical Report
• —GUI-DR Data Augmentation Pipeline
• —GUI-Perturbed Dataset
• —Model Checkpoint

Citation

@misc{wang2026guiperturbeddomainrandomizationreveals,
      title={GUI-Perturbed: Domain Randomization Reveals Systematic Brittleness in GUI Grounding Models}, 
      author={Yangyue Wang and Harshvardhan Sikka and Yash Mathur and Tony Zhou and Jinu Nyachhyon and Pranav Guruprasad},
      year={2026},
      eprint={2604.14262},
      archivePrefix={arXiv},
      primaryClass={cs.LG},
      url={https://arxiv.org/abs/2604.14262}, 
}

@online{training_on_gui_perturbed_technical_report_2026,
  title   = {Training on GUI-Perturbed: Why More Data Isn’t Enough},
  author  = {Yangyue Wang and Harshvardhan Sikka and Yash Mathur and Tony Zhou and Jinu Nyachhyon and Pranav Guruprasad},
  year    = {2026},
  url     = {www.fig.inc/blog/training-on-gui-perturbed-why-more-data-isnt-enough/},
  note    = {Part 3: Finetuning Experiments}
}