p-doom/idm

<div align="center"> <img class="dark:hidden" src="pdoomlogoblacktransparent.png" width="50%" alt="p(doom)" /> <img class="hidden dark:block" src="pdoomlogowhitetransparent.png" width="50%" alt="p(doom)" /> <br><br> <a href="https://www.pdoom.org/"><img alt="Homepage" src="https://img.shields.io/badge/Homepage-p%28doom%29-white?logo=home&logoColor=black"/></a> <a href="https://huggingface.co/p-doom"><img alt="Hugging Face" src="https://img.shields.io/badge/%F0%9F%A4%97%20Hugging%20Face-p--doom-ffc107?color=ffc107&logoColor=white"/></a> <a href="https://discord.gg/G4JNuPX2VR"><img alt="Discord" src="https://img.shields.io/badge/Discord-p%28doom%29-7289da?logo=discord&logoColor=white&color=7289da"/></a> <a href="https://github.com/p-doom"><img alt="GitHub" src="https://img.shields.io/badge/GitHub-p--doom-24292e?logo=github&logoColor=white"/></a> <a href="https://github.com/p-doom/inverse-dynamics-model"><img alt="Code" src="https://img.shields.io/badge/Code-inverse--dynamics--model-24292e?logo=github&logoColor=white"/></a> <a href="https://twitter.com/probdoom"><img alt="Twitter Follow" src="https://img.shields.io/badge/Twitter-prob_doom-white?logo=x&logoColor=white"/></a> </div>

<br>

Inverse Dynamics Model for Action-Annotating Screencasts

We present an inverse dynamics model that predicts user input actions from short windows of screen recordings. Given 10 consecutive screenshots, it emits the key presses, mouse clicks, cursor movements and scroll events that are visually implied by the frames.

Please refer to the blog post for details and experiments.

Summary

• —Base model: `Qwen/Qwen3-VL-8B-Instruct`
• —Training data: `crowd-cast` recordings
• —Training method: LoRA on language and vision modules, merged after training
• —Input: 10 screenshots sampled at 5 FPS
• —Output: sparse JSON action list
• —Eval: `p-doom/idm-eval-set`
• —Code: `p-doom/inverse-dynamics-model`

IDM-eval results

All models are evaluated on our eval set, visibility-filtered to visible + inferable actions. MM R² and MM cos_mean include missed MouseMove frames as zero predictions; MM cov. is MouseMove recall.

Interpretation: the main gap is dense temporal coverage. Off-the-shelf VLMs under-emit MouseMove actions, and the all-GT MouseMove metrics penalize these misses.

Input Format

Provide one chat message with 10 images sampled at 5 FPS. Each image should be preceded by a text label:

Frame F00: <image>
Frame F01: <image>
...
Frame F09: <image>

The frame labels are text anchors in the message, not labels rendered into the image pixels.

Output Format

The model emits only a JSON array:

[
  {"frame": "F02", "type": "MouseMove", "details": "120,45"},
  {"frame": "F03", "type": "MouseClick", "details": "Left"},
  {"frame": "F05", "type": "KeyPress", "details": "Cmd+S"},
  {"frame": "F07", "type": "MouseScroll", "details": "-150"}
]

Action types:

• —KeyPress: key name with modifiers, e.g. Cmd+S, Return, A
• —MouseClick: Left, Right, or Middle
• —MouseMove: normalized dx,dy, where 1000 is a full screen-width or screen-height traversal
• —MouseScroll: normalized signed scroll magnitude

Frame attribution: if an effect first appears between F_K and F_{K+1}, report the action on F_K, the last pre-action frame.

Related Releases

• —`p-doom/AGI-CAST-0.6k`: source AGI-CAST screencasts
• —`p-doom/AGI-CAST-idm-actions`: AGI-CAST action annotations generated with this model
• —`p-doom/idm-eval-set`: manually verified IDM evaluation clips

Limitations

• —The model was trained on macOS clips and can confuse OS-specific shortcuts such as Cmd vs Ctrl.
• —Labels are inferred from pixels, so actions with no visual evidence can be missed or hallucinated.
• —Fine-grained timing, cursor movement magnitude and scroll magnitude can be noisy.