chakra-labs/pango-sample

1---2language:3- en4tags:5- computer-use6pretty_name: Real-World Computer Use Agent Training Data7---8 9# Pango Sample: Real-World Computer Use Agent Training Data10 11**Pango** represents **P**roductivity **A**pplications with **N**atural **G**UI **O**bservations and trajectories.12 13## Dataset Description14 15This dataset contains authentic computer interaction data collected from users performing real work tasks in productivity applications. The data was collected through [Pango](https://pango.so), a crowdsourced platform where users are compensated for contributing their natural computer interactions during actual work sessions.16 17![Pango Screenshot](images/pango_1.png)18 19## Motivation20 21Current Computer Use Agent (CUA) training datasets face several limitations:22- **Scale constraints**: Existing datasets like Mind2Web (2,350 tasks), GUI-World (12,000 videos), and OSWorld (369 tasks) provide limited coverage23- **Artificial contexts**: Most demonstrations are scripted rather than authentic work sessions24- **Distribution gaps**: Performance drops significantly when agents encounter interfaces outside their training distribution25- **Missing error patterns**: Academic datasets typically exclude "failed" interactions, removing important recovery behaviors26 27This dataset addresses these limitations by capturing real users performing genuine work tasks, providing natural interaction patterns, error recovery sequences, and diverse problem-solving approaches.28 29## Data Collection Methodology30 31Data is collected through a Chrome extension that records user interactions during structured "quests" in target applications:32- **Applications**: Google Sheets, Google Slides, Figma, Canva (more coming soon)33- **User base**: Global contributor network across 180+ countries34- **Task context**: Authentic work sessions (financial analysis, presentation creation, design work, etc.)35- **Compensation**: Users are paid based on session length and data quality36 37## Dataset Structure38 39Each record contains:40- `id`: Unique session identifier41- `video_url`: Screen recording of the interaction session42- `input_metadata`: Structured JSON containing granular interaction events43- `input_metadata_preview_url`: Link to interactive data viewer for the `input_metadata`44- `task_description`: User-provided description of what they were doing45- `quest_type`: Application category (Sheets, Slides, Figma, Canva)46- `profession`: User's professional background47- `synthetically_generated_instruction`: Synthetically generated task instruction for training purposes. Represents the context of the full task.48- `synthetically_generated_thought_metadata`: (Beta) Synthetically generated thoughts for each user step. Represents the thought of the current step. Available by request.49- `artifact_history_preview_url`: Link to the full history of the data artifact. Currently only Canva data. Useful for code generation in addition to pure CUA. 50 51### Input Metadata Schema52 53The `input_metadata` field contains timestamped interaction events with the following structure:54 55```json56{57  "relative_timestamp_ms": 1028,58  "type": "click",59  "x": 186.0,60  "y": 62.445,61  "button": "button_left",62  "screenshot_url": "https://...",63  "click_count": 164}65```66 67 68**Key fields:**69- `relative_timestamp_ms`: Milliseconds since session start70- `type`: Event type (click, input, key_press, mouseover_start, mouseover_end, drag_start, drag_end, scroll)71- `x,y`: Screen coordinates (normalized for display resolution)72- `screenshot_url`: URL to corresponding interface screenshot73- `text`: Text content for input events - only available on input events74- `key_codes`: Keyboard key identifier (DOM KeyboardEvent codes) - only available on key_press events75 76### Previewing the Data77 78While the `video_url` is provided for completeness, it is often clunky to interact with `webm` files and it's impractical to leverage these for most use cases. 79 80The `input_metadata` is a much more convenient way to interact with the data, however it is not as easy to visualize. To solve this, we built an interactive preview tool, which takes the `input_metadata` and renders it frame-by-frame with corresponding actions. 81 82This can be found [here](https://admin.pango.so/actions-preview?url=https://pango-service-production.s3.us-east-1.amazonaws.com/finalized/bc50d07a-d11a-4c9c-9c5a-e5fdb6e85b5f/actions_v2.json) and it takes any valid `input_metadata` URL as input.83 84![Preview Tool Screenshot](images/preview_tool.png)85 86## Thought Metadata (Beta)87 88An additional field, `synthetically_generated_thought_metadata`, is included to provide synthetically generated thoughts for each user step. This field is designed to enhance the dataset's utility for training reasoning VLMs like [UI-TARS 1.5](https://huggingface.co/ByteDance-Seed/UI-TARS-1.5-7B). It is not to be confused with `synthetically_generated_instruction`, which is the context of the full task. This field is available by request. 89 90**Step Generation and Aggregation**91 92To create `thought_metadata`, we begin with the `input_metadata` where each row represents an individual user action. As a first stage, we aggregate actions into steps, where each step represents either a single action or a collection of actions.93 94**Batch Processing Strategy**95 96We partition the steps into batches with the following parameters:97 98$$99\alpha = 7, \quad \beta = 15, \quad \gamma = 15100$$101 102where:  103- \\(\alpha\\) (**pre_window_size**): Number of steps preceding the target step used for context  104- \\(\beta\\) (**post_window_size**): Number of subsequent steps used for context  105- \\(\gamma\\) (**batch_size**): Interval between target steps for thought generation. i.e. if \\(\gamma = 15\\), then for every 15 steps, we generate a thought.106 107The \\(\gamma = 15\\) prevents overlapping thoughts and ensures that thoughts generated in earlier batches are considered completed when generating subsequent thoughts.108 109**LLM Usage**110 111The processed step batches are fed to GPT-4o with its vision API, using high image detail settings. Each thought generation process consumes approximately 30,000 input tokens.112 113## Quality Assurance114 115Data quality is maintained through:116- Automated filtering of invalid interactions and privacy-sensitive content117- Quality scoring based on task coherence and completion patterns118- Compensation algorithms that reward genuine engagement119- Differential privacy techniques to prevent individual behavior reconstruction120 121## Use Cases122 123This dataset is designed for:124- Training computer use agents on authentic interaction patterns125- Studying human-computer interaction behaviors across diverse populations126- Developing more robust GUI automation systems127- Research on temporal reasoning and error recovery in sequential decision-making128 129## Ethical Considerations130 131- All users provide informed consent for data collection and redistribution132- Privacy-sensitive content is automatically filtered133- Compensation ensures fair value exchange for user contributions134- Data collection follows ethical guidelines for crowdsourced research135 136## Data Scale and Growth137 138The dataset is continuously growing through ongoing collection:139- Planned: Scaling to 100,000+ hours over 2025140 141## Citation142 143If you use this dataset in your research, please cite:144 145```bibtex146@dataset{pango2025,147  title={Pango: Real-World Computer Use Agent Training Data},148  author={Chakra Labs},149  year={2025},150  url={https://huggingface.co/datasets/chakra-labs/pango}151}152```153 154## Contact155 156For access to the full dataset or collaboration opportunities, please contact [Chakra Labs](mailto:nirmal@chakra-labs.com).157 158