Apppublic

zorba111/ui-coordinates-finder

sourceHugging Faceupdated 2y agoView on Hugging Face

0likes

App README

OmniParser: Screen Parsing tool for Pure Vision Based GUI Agent

![arXiv](https://arxiv.org/abs/2408.00203) ![License](https://opensource.org/licenses/MIT)

OmniParser is a comprehensive method for parsing user interface screenshots into structured and easy-to-understand elements, which significantly enhances the ability of GPT-4V to generate actions that can be accurately grounded in the corresponding regions of the interface.

News

—[2024/10] Both Interactive Region Detection Model and Icon functional description model are released! Hugginface models
—[2024/09] OmniParser achieves the best performance on Windows Agent Arena!

Install

Install environment:

python

conda create -n "omni" python==3.12
conda activate omni
pip install -r requirements.txt

Then download the model ckpts files in: https://huggingface.co/microsoft/OmniParser, and put them under weights/, default folder structure is: weights/icondetect, weights/iconcaptionflorence, weights/iconcaption_blip2.

Finally, convert the safetensor to .pt file.

python

python weights/convert_safetensor_to_pt.py

Examples:

We put together a few simple examples in the demo.ipynb.

Gradio Demo

To run gradio demo, simply run:

python

python gradio_demo.py

📚 Citation

Our technical report can be found here. If you find our work useful, please consider citing our work:

@misc{lu2024omniparserpurevisionbased,
      title={OmniParser for Pure Vision Based GUI Agent},
      author={Yadong Lu and Jianwei Yang and Yelong Shen and Ahmed Awadallah},
      year={2024},
      eprint={2408.00203},
      archivePrefix={arXiv},
      primaryClass={cs.CV},
      url={https://arxiv.org/abs/2408.00203},
}

title: Ui Element Coordinates Finder emoji: 🏢 colorFrom: pink colorTo: red sdk: gradio sdkversion: 5.4.0 appfile: app.py pinned: false license: mit

Check out the configuration reference at https://huggingface.co/docs/hub/spaces-config-reference