p-doom/AGI-CAST-idm-actions

<div align="center"> <img class="dark:hidden" src="pdoomlogoblacktransparent.png" width="50%" alt="p(doom)" /> <img class="hidden dark:block" src="pdoomlogowhitetransparent.png" width="50%" alt="p(doom)" /> <br><br> <a href="https://www.pdoom.org/"><img alt="Homepage" src="https://img.shields.io/badge/Homepage-p%28doom%29-white?logo=home&logoColor=black"/></a> <a href="https://huggingface.co/p-doom"><img alt="Hugging Face" src="https://img.shields.io/badge/%F0%9F%A4%97%20Hugging%20Face-p--doom-ffc107?color=ffc107&logoColor=white"/></a> <a href="https://discord.gg/G4JNuPX2VR"><img alt="Discord" src="https://img.shields.io/badge/Discord-p%28doom%29-7289da?logo=discord&logoColor=white&color=7289da"/></a> <a href="https://github.com/p-doom"><img alt="GitHub" src="https://img.shields.io/badge/GitHub-p--doom-24292e?logo=github&logoColor=white"/></a> <a href="https://twitter.com/probdoom"><img alt="Twitter Follow" src="https://img.shields.io/badge/Twitter-prob_doom-white?logo=x&logoColor=white"/></a> </div>

<br>

AGI-CAST IDM Actions

We release model-generated keyboard, mouse-click, cursor-movement and scroll annotations for the AGI-CAST-0.6k screencast dataset.

The labels were generated with `p-doom/idm`, an inverse dynamics model trained to recover user input actions from short windows of screen recordings.

Please refer to the AGI-CAST blog post and the IDM blog post for more context.

Code

These labels were generated with `p-doom/idm`. The labeling code is available at `p-doom/inverse-dynamics-model`.

Dataset Summary

AGI-CAST contains long-horizon recordings of p(doom) members doing real AGI research work. This companion dataset adds sparse action labels to those screencasts: for each 10-frame window sampled at 5 FPS, the IDM predicts the user actions visible in that window.

The action space is:

• —KeyPress: key name with modifiers, e.g. Cmd+S, Return, A
• —MouseClick: Left, Right, or Middle
• —MouseMove: normalized relative cursor displacement dx,dy
• —MouseScroll: normalized signed scroll magnitude

These are weak labels produced by a model, not human-verified ground truth.

Key Features

• —Source Dataset: `p-doom/AGI-CAST-0.6k`
• —Total Duration: 646.21 hours
• —Source Videos: 147 screencasts
• —Action Labels: 6.89 million predicted actions
• —Temporal Resolution: 5 FPS
• —Window Size: 10 frames, corresponding to 2 seconds

Dataset Statistics

Action Counts

Contributor Breakdown

Data Files

The dataset is released in three views:

• —actions: one row per predicted action. This is the most convenient view for training or action-frequency analysis.
• —chunks: one row per 10-frame window, including windows with no predicted actions.
• —videos: one row per source video with lightweight alignment metadata.

The files are stored as Parquet because the release is fundamentally tabular and contains 6.9 million action rows. Parquet keeps the repository compact, preserves nested action lists in chunks, supports column-selective loading, and works directly with Hugging Face Datasets. The original per-video JSON outputs were converted because they contained internal file paths and are less efficient for this scale.

Schema

actions

Each row is a single predicted action.

chunks

Each row is a 10-frame window. The actions column is a nested list with the action fields above. Empty windows are retained with n_actions = 0.

videos

Each row contains source-video alignment metadata: FPS, number of extracted frames, number of windows, duration, prompt variant, and model repo.

Loading

from datasets import load_dataset

actions = load_dataset("p-doom/AGI-CAST-idm-actions", "actions", split="train")
chunks = load_dataset("p-doom/AGI-CAST-idm-actions", "chunks", split="train")
videos = load_dataset("p-doom/AGI-CAST-idm-actions", "videos", split="train")

Label Generation

Labels were generated by running `p-doom/idm` over AGI-CAST videos sampled at 5 FPS. Each model call sees a 10-frame window and emits a sparse JSON array of actions.

The frame convention follows the IDM: if an effect first appears between F_K and F_{K+1}, the action is attributed to F_K, the last pre-action frame.

Limitations

• —Labels inherit the failure modes of the IDM.
• —Some actions are visually ambiguous or not recoverable from frames alone.
• —Fine-grained key timing, cursor movement magnitude and scroll magnitude can be noisy.
• —To align labels with pixels, pair these annotations with the corresponding videos in `p-doom/AGI-CAST-0.6k`.

Attribution

If you use AGI-CAST or these IDM action labels in your research, please cite the AGI-CAST release:

@article{srambical2025agi-cast,
  author = {Srambical, Franz},
  title = {AGI-CAST: Making Agents Work Like Humans},
  journal = {p(doom) blog},
  year = {2025},
  note = {https://pdoom.org/agi_cast.html}
}