nnc-001/gtasa-01

GTASA-01: Multi-Actor Video Corpus with Perfect Spatiotemporal Annotations

GTASA-01 is the sample corpus released with the ICLR 2026 Tiny Paper **GEST-Engine: Controllable Multi-Actor Video Synthesis with Perfect Spatiotemporal Annotations**.

The corpus contains 398 procedurally generated multi-actor stories produced by the GEST-Engine, each accompanied by a Graph of Events in Space and Time (GEST) specification, an engine-rendered RGB video with dense spatiotemporal annotations, and — for comparison — videos produced by VEO 3.1 and WAN 2.2 from the same textual prompt.

Corpus statistics

Directory structure

Each story folder is named after its generation configuration (e.g., classroom_max2actors_max1regions_2action_chains_b9d2ff0d) and follows this layout:

<story_folder>/
├── texts.json                         # GPT-4o query + refined natural-language description
├── veo3-1.mp4                         # VEO 3.1 video from the refined description
├── wan2.2.mp4                         # WAN 2.2 video from the refined description
│
├── detailed_graph/
│   └── take1/
│       ├── detail_gest.json           # the input GEST specification
│       └── proto-graph.json           # normalized-ID GEST with populated timeframes
│
└── simulations/
    └── take1_sim1/
        ├── event_frame_mapping.json   # {event_id → [startFrame, endFrame]} alignments
        │
        ├── camera1/
        │   ├── raw.mp4                # engine-rendered RGB video
        │   ├── segmentation_frames.zip     # per-frame HLSL instance segmentation masks
        │   ├── segmentation_mapping.json   # texture hash → story-level entity ID
        │   └── spatial_relations.zip       # per-frame pairwise spatial relation graphs
        │
        ├── logs/
        │   ├── clientscript.log       # MTA client-side script log
        │   └── server.log             # MTA server-side script log
        │
        └── textual_description/
            ├── engine_generated.txt   # Logger running commentary during simulation
            └── prompt.txt             # proto-language with GPT-4o instructions

File descriptions

Top-level (per story)

• —`texts.json` — Output of the two-stage text generation pipeline (proto-language + LLM refinement). Contains the GPT-4o query (the proto-language paragraph wrapped in the refinement instruction) and the refined natural-language description returned by GPT-4o (gpt-4o-2024-08-06). The refined description is what was used to prompt VEO 3.1 and WAN 2.2.
• —`veo3-1.mp4` — Video generated by VEO 3.1 using the refined description from texts.json as prompt.
• —`wan2.2.mp4` — Video generated by WAN 2.2 using the same refined description as prompt.

`detailed_graph/take1/`

• —`detail_gest.json` — The Graph of Events in Space and Time specification for this story, including actor and object Exists nodes, per-event action / entities / location / timeframe / properties, and the temporal / spatial / semantic / camera relation sections.
• —`proto-graph.json` — Intermediate transformation of the GEST used by the text generation pipeline: entity identifiers are normalized to a canonical format (e.g., a0 → actor0; spawnable IDs become id:0.0-class:mobilephone), and each event's Timeframe field is populated with the exact [startFrame, endFrame] range from the event-frame mapping.

`simulations/take1_sim1/`

• —`event_frame_mapping.json` — Exact frame-level alignment of each GEST event to its start/end frame in the rendered video ({eventId → [startFrame, endFrame]}), with FPS metadata.

`simulations/take1_sim1/camera1/`

• —`raw.mp4` — RGB video of the multi-actor simulation rendered by the engine.
• —`segmentation_frames.zip` — Per-frame instance segmentation masks produced via an HLSL shader with FNV-1a texture hashing.
• —`segmentation_mapping.json` — Mapping from texture hash values to story-level entity IDs, linking segmentation masks back to the GEST specification.
• —`spatial_relations.zip` — Per-frame pairwise spatial relation graphs (one JSON per frame). For each entity, each frame records: 3D position and rotation; camera-relative distance, horizontal and vertical angles, coarse direction bucket (front/back/left/right/above/below/combinations), and in-FOV flag; object type and model ID. Entities are tagged with their story-level storyObjectId, linking back to the input GEST. The camera state (position, lookAt, FOV, roll) is also stored per frame.

`simulations/take1_sim1/logs/`

• —`clientscript.log` — Client-side Multi Theft Auto script log.
• —`server.log` — Server-side Multi Theft Auto script log.

`simulations/take1_sim1/textual_description/`

• —`engine_generated.txt` — Running-commentary text produced by the engine's Logger during simulation, reporting actions as they execute.
• —`prompt.txt` — The proto-language paragraph (ungrammatical verb forms like sitdowns, takeouts, assembled mechanically from the proto-graph) wrapped with the instruction prompt sent to GPT-4o.

Source code

• —GEST-Engine (simulation system): github.com/ncudlenco/mta-sim
• —Procedural GEST generator and batch production orchestrator: github.com/ncudlenco/multiagent_story_system

Both repositories are tagged at v1.0-iclr2026, the exact state used to generate this corpus.

License and intellectual property notice

This dataset is released under CC BY-NC 4.0 for non-commercial academic research purposes only.

The videos in this corpus contain frames rendered by Grand Theft Auto: San Andreas (Rockstar Games / Take-Two Interactive, 2004) via the Multi Theft Auto modification framework. All in-game assets (3D models, textures, animations, environments) remain the property of their respective owners. We do not claim ownership of any Rockstar Games / Take-Two Interactive intellectual property. Use of this dataset is governed by both the CC BY-NC 4.0 license and applicable copyright law regarding the underlying game content.

Citation

If you use this corpus, please cite:

@inproceedings{cudlenco2026tiny,
  title={[Tiny Paper] {GEST}-Engine: Controllable Multi-Actor Video Synthesis with Perfect Spatiotemporal Annotations},
  author={Nicolae Cudlenco and Mihai Masala and Marius Leordeanu},
  booktitle={ICLR 2026 the 2nd Workshop on World Models: Understanding, Modelling and Scaling},
  year={2026},
  url={https://openreview.net/forum?id=uUofPYVMZH}
}

Contact

For questions or issues, please open an issue on the GEST-Engine repository or contact nicolae.cudlenco@gmail.com. </content> </invoke>