Datasetpublic

ivelin/ui_refexp

This dataset is intended for UI understanding, referring expression and action automation model training. It's based on the UIBert RefExp dataset from Google Research, which is based on the RICO dataset.

sourceHugging Facecc-by-4.0updated 4y agoView on Hugging Face

4likes22downloads

rootREADME.md

README.md134 linesDownload Raw Back to root

1---2license: cc-by-4.03task_categories:4  - image-to-text5tags:6  - ui-referring-expression7  - ui-refexp8language:9  - en10pretty_name: UI understanding11size_categories:12  - 10K<n<100K13dataset_info:14  features:15    - name: screenshot16      dtype: image17    - name: prompt18      dtype: string19    - name: target_bounding_box20      dtype: string21  config_name: ui_refexp22  splits:23    - name: train24      num_bytes: 56203726525      num_examples: 1562426    - name: validation27      num_bytes: 6039922528      num_examples: 47129    - name: test30      num_bytes: 6907396931      num_examples: 56532  download_size: 651501217633  dataset_size: 69151045934---35 36# Dataset Card for UIBert37 38## Dataset Description39 40- **Homepage:** https://github.com/google-research-datasets/uibert41- **Repository:** https://github.com/google-research-datasets/uibert42- **Paper:** https://arxiv.org/abs/2107.1373143- **Leaderboard:**44  - UIBert: https://arxiv.org/abs/2107.1373145  - Pix2Struct: https://arxiv.org/pdf/2210.0334746 47### Dataset Summary48 49This is a Hugging Face formatted dataset derived from the [Google UIBert dataset](https://github.com/google-research-datasets/uibert), which is in turn derived from the [RICO dataset](https://interactionmining.org/rico).50 51### Supported Tasks and Leaderboards52 53- UI Understanding54- UI Referring Expressions55- UI Action Automation56 57### Languages58 59- English60 61## Dataset Structure62 63- `screenshot`: blob of pixels.64- `prompt`: Prompt referring to a UI component with an optional action verb. For example "click on search button next to menu drawer."65- `target_bounding_box`: Bounding box of targeted UI components. `[xmin, ymin, xmax, ymax]`66 67### Data Splits68 69- train: 15K samples70- validation: 471 samples71- test: 565 samples72 73## Dataset Creation74 75### Curation Rationale76 77[More Information Needed]78 79### Source Data80 81#### Initial Data Collection and Normalization82 83[More Information Needed]84 85#### Who are the source language producers?86 87[More Information Needed]88 89### Annotations90 91#### Annotation process92 93[More Information Needed]94 95#### Who are the annotators?96 97[More Information Needed]98 99### Personal and Sensitive Information100 101[More Information Needed]102 103## Considerations for Using the Data104 105### Social Impact of Dataset106 107[More Information Needed]108 109### Discussion of Biases110 111[More Information Needed]112 113### Other Known Limitations114 115[More Information Needed]116 117## Additional Information118 119### Dataset Curators120 121[More Information Needed]122 123### Licensing Information124 125[More Information Needed]126 127### Citation Information128 129[More Information Needed]130 131### Contributions132 133[More Information Needed]134