contralabs/premiere-video-editing-trajectories

Creative Video-Editing Computer-Use Trajectories (Preview)

A preview release of computer-use agent trajectories from professional video-editing work in Adobe Premiere Pro (building vertical short-form social reels). Each step pairs a screenshot with a structured action and a first-person thought grounded in the editor's spoken narration as they worked, so the step-level reasoning reflects real human intent rather than a rationale written after the fact.

A sample of the human data [Contra Labs](https://contralabs.com/?utm_source=huggingface&utm_medium=dataset_card&utm_campaign=creative-video-editing&utm_content=hero) builds on demand. We are an independent human-data and creative-evaluation lab: expert evaluation, rankings, and benchmarks for AI outputs, plus custom datasets like this one. See Working with Contra Labs.

[image]

Every step is captured three ways and the three are checked against each other. The screenshot is a pre-action keyframe, so it shows the state the editor saw when deciding rather than the result of the action; the tool_call is recorded telemetry read off Premiere's own event payload; the thought comes from the editor's narration while working. Where the narration and the telemetry disagree, the telemetry wins and the prose is what gets corrected.

Motivation

Most open computer-use and GUI-agent datasets focus on short, single-goal web or OS tasks, and the step-level reasoning they carry is usually generated by a model after the fact rather than captured from the person doing the work. This preview is a small sample of a different kind of data: multi-step professional video editing, with reasoning taken from the editor's own narration as they work.

Relative to typical GUI-agent trajectories, it shows:

• —Multi-step timeline-editing workflows in Adobe Premiere Pro, not short single-goal tasks.
• —Step-level thought grounded in the editor's recorded narration, not model-synthesized rationales.
• —Executable action grounding through an Adobe MCP server (see below).
• —Tool-rich creative operations (Lumetri grading, captions, transitions, reframing) mapped to a Premiere action taxonomy.

What a recording session contains

A recording session follows an editor building one vertical social reel from a client brief. The editor imports footage and screen recordings into Premiere, lays out selects on the timeline, trims and reorders clips, reframes for a 9:16 aspect, grades with Lumetri, adds captions and transitions, balances audio, and exports. Group by session_uuid to recover a session.

Format and schema

The dataset is a single Parquet file with one row per step: 234 steps across 4 trajectories from 4 recording sessions (30 to 114 steps and 111 to 245 minutes per trajectory), recorded on macOS. Screenshots are embedded in the Parquet as a Hugging Face Image feature. Trajectory-level fields are repeated on every step so a row is self-contained; group by trajectory_uuid and order by step_index to reconstruct a trajectory. Each trajectory ends with a synthetic terminate step (action_type is terminate) marking the stop decision.

from datasets import load_dataset

ds = load_dataset("contra-labs/creative-video-editing-trajectories", split="train")
row = ds[0]
print(row["instruction"])
print(row["thought"])
row["image"].show()

Per-step fields:

Trajectory-level fields (denormalized, repeated on every step):

Reasoning provenance: thought, observation, and reflection are model-augmented from the editor's spoken narration. The screenshots, the recorded action, and the pointer coordinates are the editor's real execution. code is a pyautogui-style rendering of the action; tool_call and execution_paths are replay mappings the pipeline attached (and in some cases corrected), not raw captures.

Executing the trajectories (Adobe Premiere MCP)

The Premiere steps are meant to be executable, not only observed. Where a deterministic mapping exists, a step's execution_paths gives up to three ways to perform the same operation:

• —mcp_tool: a tool from the Adobe Premiere MCP server, with its name and argument_keys (for example add_media_to_sequence, set_clip_start_end_times, apply_effect, append_video_transition, set_video_clip_properties, import_media, export_sequence).
• —keyboard_shortcut: the OS-specific hotkey, split into macos and windows.
• —menu_path: the Premiere menu navigation, for example ["Sequence", "Apply Video Transition"].

preferred_execution names which to try first, favoring a keyboard shortcut or tool call over a coordinate-based mouse click. Not every timeline edit has an MCP tool; where one does not exist yet, the step carries a menu path or shortcut, or code alone.

[image]

The mcp_tool names map onto adb-mcp, an MCP server for Adobe applications:

• —Upstream: mikechambers/adb-mcp (Photoshop, Premiere, InDesign, After Effects, Illustrator).
• —Fork used here: JasonObeid/adb-mcp, which extends the tool set (clip placement and trimming, effects, transitions, clip properties, markers, and export) so timeline edits are reproducible as tool calls.

Building on current computer-use research

We mainly follow the trajectory format of the AgentNet computer-use corpus (and related work such as AgentTrek), so the data is familiar to practitioners and interoperable with existing computer-use tooling rather than introducing another bespoke schema. We also carry over an idea from ToolCUA: a single goal can be reached through multiple modalities (atomic GUI actions or higher-level tool calls), which we represent per step as execution_paths with a preferred_execution.

The shared per-step pattern is a screenshot, a first-person observation and thought, a structured action (an action-type label plus a pyautogui-style action code), and a post-step reflection.

Onto that base we add a small number of fields, motivated by tool-rich creative work and by executable deployment:

Intended use

• —Exploring what multi-step professional video-editing trajectories look like as training and evaluation data.
• —Studying how narration-grounded reasoning differs from model-synthesized rationales.
• —Previewing the trajectory schema, execution paths, and tool-call grounding for creative-professional work.

Limitations and scope

This is a preview rather than a training-scale corpus. It covers a single vertical (vertical-reel editing in Adobe Premiere Pro) on macOS, and is intended for evaluation and qualitative study rather than large-scale training.

Working with Contra Labs

Contra Labs is an independent human-data and creative-evaluation lab, backed by a network of verified creative and domain experts. This dataset is one example of our work.

We partner with AI teams on:

• —Evaluation, rankings, and benchmarks. Expert human judgment on model outputs across text, image, video, audio, UI, and multi-modal work, scored for quality, style, and brand fit.
• —Custom dataset creation. Computer-use trajectories, pairwise preference data, and evaluation scores, custom built to your domain, schema, and difficulty. Data can be delivered as agentic message arrays (chat-format SFT data), reasoning-only sequences, network-request logs, preference pairs for DPO/RLHF, reward-model scores, or raw trajectories, matched to your pipeline's expected schema.

To commission an evaluation or dataset for your domain, reach out through contralabs.com or email partnerships@contralabs.com.

Provenance and consent

The editors who performed the work took part with consent. Each screenshot was reviewed to mask the operator's recording setup and account identity: the screen-recorder window and its live camera, account avatars and names, profile photos, usernames, device names, and private folder listings. The footage being edited is retained as shown, so the on-screen subjects of the video (the people appearing in the reel) remain visible and their spoken words appear in the captions and transcript. Brand and tool names appear as part of the work shown.

References

• —AgentNet computer-use dataset: huggingface.co/datasets/xlangai/AgentNet. OpenCUA: Open Foundations for Computer-Use Agents, arXiv:2508.09123.
• —ToolCUA: Towards Optimal GUI-Tool Path Orchestration for Computer Use Agents. arXiv:2605.12481.
• —AgentTrek: Agent Trajectory Synthesis via Guiding Replay with Web Tutorials. ICLR 2025. arXiv:2412.09605.
• —Adobe MCP server (adb-mcp): mikechambers/adb-mcp; fork used here: JasonObeid/adb-mcp.

Citation

@misc{contra_labs_creative_video_editing_trajectories_2026,
  title        = {Creative Video-Editing Computer-Use Trajectories (Preview)},
  author       = {Contra Labs},
  year         = {2026},
  howpublished = {Hugging Face Datasets},
  note         = {Preview release}
}

License

Released under CC-BY-4.0. Free to use with attribution to Contra Labs.