VideoGUI/VideoGUI-Action

Dataset Card for VideoGUI-Action

This is the Action execution part of VideoGUI benchmark.

We uploaded the full metadata, recording at Google Drive.

Data Fields

• —"app" (str): The name of the software being used.
• —"task_id" (str): A unique identifier for each full task.
• —"screenshot_start" (image): The screenshot image representing the state at the beginning of the action.
• —"action_type" (str): The type of action performed e.g., (right) click, drag, type, scroll
• —"action_narration" (str): A textual description of the action, narrating what happens during the action.
• —"start_position" (list[int]): The starting position of the action on the screen, represented as a sequence of integers (e.g., coordinates).
• —"end_position" (list[int]): The ending position of the action on the screen, represented as a sequence of integers (e.g., coordinates).
• —"scroll_value" (float): The value of the scroll action if applicable, indicating the amount of scrolling performed.
• —"action_sequence" (str): The sequence of actions taken, represented as a string for easier understanding and processing.
• —"subtask_id" (int): A unique identifier (ranking) for the subtask within the full task.
• —"subtask_query" (str): The query or input related to the subtask, which may guide the actions sequence taken.
• —"action_id" (int): A unique identifier (ranking) for the specific action performed within the task.