Datasetpublic

neulab/Mind2Web_train_llava

Mind2Web training set for the paper: Harnessing Webpage Uis For Text Rich Visual Understanding 🌐 Homepage | 🐍 GitHub | 📖 arXiv Introduction We introduce MultiUI, a dataset containing 7.3 million samples from 1 million websites, covering diverse multi- modal tasks and UI layouts. Models trained on MultiUI not only excel in web UI tasks—achieving up to a 48% improvement on VisualWebBench and a 19.1% boost in action accuracy on a web agent dataset Mind2Web—but… See the full description on the dataset page: https://huggingface.co/datasets/neulab/Mind2Web_train_llava.

sourceHugging Faceodc-byupdated 2y agoView on Hugging Face

2likes120downloads

Dataset Card

Mind2Web training set for the paper: Harnessing Webpage Uis For Text Rich Visual Understanding

🌐 Homepage | 🐍 GitHub | 📖 arXiv

Introduction

We introduce MultiUI, a dataset containing 7.3 million samples from 1 million websites, covering diverse multi- modal tasks and UI layouts. Models trained on MultiUI not only excel in web UI tasks—achieving up to a 48% improvement on VisualWebBench and a 19.1% boost in action accuracy on a web agent dataset Mind2Web—but also generalize surprisingly well to non-web UI tasks and even to non-UI domains, such as document understanding, OCR, and chart interpretation.

Contact

—Junpeng Liu: jpliu@link.cuhk.edu.hk
—Xiang Yue: xyue2@andrew.cmu.edu

Citation

If you find this work helpful, please cite out paper:

@misc{liu2024harnessingwebpageuistextrich,
      title={Harnessing Webpage UIs for Text-Rich Visual Understanding}, 
      author={Junpeng Liu and Tianyue Ou and Yifan Song and Yuxiao Qu and Wai Lam and Chenyan Xiong and Wenhu Chen and Graham Neubig and Xiang Yue},
      year={2024},
      eprint={2410.13824},
      archivePrefix={arXiv},
      primaryClass={cs.CV},
      url={https://arxiv.org/abs/2410.13824}, 
}