contralabs/descript-video-editing-trajectories

Descript Video-Editing Computer-Use Trajectories (Preview)

This is a preview release of computer-use trajectories from experienced video editors working through client-style editing briefs in Descript: cutting vertical short-form social reels from source footage. Each session is a long edit, about two hours and a few hundred steps, and the editor's spoken narration was recorded while they worked and used to ground the step-level reasoning. Most open GUI-agent datasets cover short, single-goal web or OS tasks with model-generated reasoning. This one covers longer, multi-step editing work with reasoning grounded in the editor's live narration.

[image]

From Contra Labs

This dataset is a sample of the human data Contra Labs builds on demand. We are an independent human-data and creative-evaluation lab, backed by a network of verified creative and domain experts. We produce expert evaluations, rankings, benchmarks, and custom computer-use datasets like this one for AI teams.

Other video datasets from us on Hugging Face:

• —Adobe Premiere editing trajectories. The native desktop-app editing sibling to this Descript set, with the same schema, so the two combine.
• —Video detail annotation. Dense human descriptions and annotations of video content.

Browse all our datasets on Hugging Face.

Want data like this for your domain? We build computer-use trajectories, pairwise preference data, and expert evaluation scores to your schema and difficulty. Delivery formats include chat-format SFT arrays, reasoning-only sequences, preference pairs for DPO/RLHF, reward-model scores, and raw trajectories. Reach out at contralabs.com or partnerships@contralabs.com. See Working with Contra Labs for more.

What's in it

A subsample of 3 full editing sessions, one finished vertical reel each, as 803 steps in a single Parquet file (one row per step):

Notable properties:

• —Reasoning grounded in the editor's narration. The thought, observation, and reflection are written from the editor's spoken narration and the recorded edit on each step, rather than a rationale composed afterward from the finished trace.
• —Actions grounded in the app's edit-commit log. Each action_type and tool_call is read from Descript's own document-commit stream, so the recorded operation is independent of the pixels.
• —Browser-executable targets. Many interactions carry the DOM locator (playwright_locator) the operator hit, for deterministic replay, alongside the pointer coordinate.
• —Varied creative operations. Scene splits, trims, filler-word removal, transitions, overlay layers, color grade, titles, and audio balancing, each mapped to a Descript action taxonomy rather than flattened to raw clicks.
• —Curated and de-identified. Pruned to a single successful edit path per session, with the operator's recording setup and account identity masked in every frame.

The task (study setup)

Each editor received a client-style brief and was asked to edit and export a short vertical Instagram Reel in Descript from a set of source clips, working in Chrome. The task is stated at increasing specificity in the instruction, natural_language_task, and actual_task fields: review the brief, import the raw MP4 assets, create compositions from the source files, assemble a rough-draft cut, then shape it into a finished reel. Shaping the reel meant cutting the transcript to tell the story, splitting and trimming scenes, removing filler words, pinning overlay layers, adding transitions and titles, grading and balancing audio, and exporting. The editors narrated their intent aloud as they worked, and that narration is the source of the step-level reasoning.

Quickstart

from datasets import load_dataset

ds = load_dataset("contra-labs/descript-video-editing-trajectories", split="train")
row = ds[0]
print(row["instruction"])
print(row["thought"])
row["image"].show()

Group by trajectory_uuid and order by step_index to reconstruct a session; group by session_uuid to recover the recording session. Each trajectory ends with a synthetic terminate step (action_type is terminate). Screenshots are embedded as a Hugging Face Image feature, and trajectory-level fields are repeated on every row so a row is self-contained.

Per-step fields:

Trajectory-level fields (denormalized, repeated on every step): instruction, natural_language_task, and actual_task (the task at increasing specificity); platform; and applications_used, step_count, and duration_ms (session summary).

Intended use

• —Exploring what long, multi-step professional video-editing trajectories look like as training and evaluation data.
• —Studying how narration-grounded reasoning differs from model-generated rationales.
• —Previewing the trajectory schema, browser execution paths, and commit-grounded tool-call data for creative-professional work.

This is a preview rather than a training-scale corpus. It covers a single vertical (vertical-reel editing in Descript) and is intended for evaluation and qualitative study rather than large-scale training.

Methodology

The rest of this card covers methodology: how the traces are reconstructed, grounded, and curated. Skip it unless you need the internals.

Reconstructing a step

Descript is a browser app with no scripting plugin, so there is no editor-side action log to read. The action ground truth is recovered from the app's own collaborative edit-commit stream: every edit the operator makes is committed to Descript's document graph with a typed operation and a delta. That commit log is what each step's action_type and tool_call are built from, which makes the recorded operation independent of the pixels and of the narration. Each of the other fields is recovered from a signal the rollout captured:

[image]

Every step is captured three ways, and the three are checked against each other. The screenshot is a pre-action keyframe, so it shows the state the editor saw when deciding rather than the result of the action. The tool_call is recorded telemetry read from the app's commit log. The thought comes from the editor's narration while working. Where the narration and the telemetry disagree, the telemetry wins and the prose is corrected to match.

Reasoning provenance. thought, observation, and reflection are model-augmented from the editor's spoken narration and the recorded edit commit, written against the step's own screenshot. The reflection is backward-looking in the AgentNet L2 style: it reads the current frame, which under the pre-action keyframe convention is the previous action's result, to judge whether that action landed and to catch errors, rather than predicting a future frame. The screenshots, the recorded edit operations, the DOM locators, and the pointer coordinates are the editor's real execution. code is a pyautogui-style rendering of the action; tool_call and execution_paths are built from recorded telemetry rather than inferred from the image.

Replaying the trajectories (browser locators)

Descript runs in the browser and has no MCP server, so the deterministic replay path is the DOM. Where the recorder captured the element the operator interacted with, execution_paths carries a playwright_locator, the same locator syntax Playwright uses, and preferred_execution names it as the path to try first, ahead of a coordinate click:

{"playwright_locator": "getByTestId('timestrip').getByText('0:05.5', { exact: true })",
 "keyboard_shortcut": null, "menu_path": null}

Not every step has one. Keyboard-driven edits carry no pointer target, and some interactions resolve to a container rather than a named control; those steps fall back to code and the pointer coordinates.

Building on current computer-use research

We mainly follow the trajectory format of the AgentNet computer-use corpus (and related work such as AgentTrek), so the data is familiar to practitioners and interoperable with existing computer-use tooling rather than a new bespoke schema. We also borrow the framing from ToolCUA that a single goal can be reached at more than one level: a pixel action, or a higher-level structured operation. Each step carries both the recorded pixel action (with coordinates and code) and a tool_call naming the Descript operation it performed. We are more modest than the Adobe Premiere sibling on the multiple-execution-paths point: Descript is a browser app with no tool API, so execution_paths here carries only the DOM playwright_locator, a more deterministic form of the same click, not the keyboard-shortcut, menu-path, and MCP-tool alternatives a native application exposes.

AgentNet structures its per-step reasoning as observation, thought, and action, with the screenshot captured as a pre-action keyframe and a reflective, error-aware trace folded into the thought. We keep the same shape (a pre-action screenshot, a first-person observation and thought, and a structured action with a pyautogui-style code) and surface the reflection as its own field in AgentNet's backward-looking style. One place we deliberately differ: this release is curated to a single successful edit path per session, so it does not keep the incorrect and redundant steps that AgentNet retains for their recover-from-error signal.

On top of that base we add a few fields, motivated by tool-rich creative work and executable deployment:

Working with Contra Labs

Contra Labs is an independent human-data and creative-evaluation lab, backed by a network of verified creative and domain experts. This dataset is one example of our work. We partner with AI teams on:

• —Evaluation, rankings, and benchmarks. Expert human judgment on model outputs across text, image, video, audio, UI, and multi-modal work, scored for quality, style, and brand fit.
• —Custom dataset creation. Computer-use trajectories, pairwise preference data, and evaluation scores, built to your domain, schema, and difficulty. Data can be delivered as agentic message arrays (chat-format SFT data), reasoning-only sequences, network-request logs, preference pairs for DPO/RLHF, reward-model scores, or raw trajectories.

To commission an evaluation or dataset for your domain, reach out through contralabs.com or email partnerships@contralabs.com.

Provenance and consent

The editors who performed the work took part with consent. Each screenshot was reviewed to mask the operator's recording setup and account identity: the screen-recorder window and its live camera, account avatars and names, profile photos, usernames, device names, private notes, and coarse location. The footage being edited is retained as shown, so the on-screen subjects of the video (the people appearing in the reel) remain visible and their spoken words appear in the captions and transcript. Brand and tool names appear as part of the work shown.

References

• —AgentNet computer-use dataset: huggingface.co/datasets/xlangai/AgentNet. OpenCUA: Open Foundations for Computer-Use Agents, arXiv:2508.09123.
• —ToolCUA: Towards Optimal GUI-Tool Path Orchestration for Computer Use Agents. arXiv:2605.12481.
• —AgentTrek: Agent Trajectory Synthesis via Guiding Replay with Web Tutorials. ICLR 2025. arXiv:2412.09605.
• —Sibling dataset (Adobe Premiere): Creative Video-Editing Computer-Use Trajectories.

Citation

@misc{contra_labs_descript_video_editing_trajectories_2026,
  title        = {Descript Video-Editing Computer-Use Trajectories (Preview)},
  author       = {Contra Labs},
  year         = {2026},
  howpublished = {Hugging Face Datasets},
  note         = {Preview release}
}

License

Released under CC-BY-4.0. Free to use with attribution to Contra Labs.