zhush/incantation-elden-ring-scenes

Incantation Elden Ring Combat Captions

[Paper](https://arxiv.org/abs/2605.18601) | [Project page](https://matrixteam-ai.github.io/pages/Incanation/) | [GitHub](https://github.com/zhushangwen/Incantation)

Preview subset. This repository is a public preview and reference subset of the Incantation dataset. It documents the data format, annotation style, and initial training material used by the project. It should not be interpreted as the final dataset, the complete benchmark, or the full data scale used by the paper.

This dataset contains manually collected Elden Ring combat clips paired with structured, action-oriented metadata for player-boss interactions. The current preview release focuses on the Tarnished player character fighting Margit and Crucible Knight-style enemies.

The data is intended to support research on controllable video generation, interactive world models, game-agent training, video-language grounding, and multi-entity action understanding. In our internal tests, this preview subset is already sufficient to train a reasonably strong interactive video generation model. The full paper-scale results, however, use a larger dataset and training pipeline.

Release Scope

This repository is a preview subset, not the complete Incantation dataset. It makes the data format, annotation style, and initial training material available for reproducibility checks, prototyping, and community reference. The paper, project page, and inference code are now public.

Planned future releases include:

• —Larger-scale Elden Ring data covering more bosses and richer action distributions.
• —The King of Fighters data used for cross-world experiments.
• —Model checkpoints and benchmark scripts, subject to release constraints.

Manual Collection Statement

All clips in this release were collected, filtered, organized, and reviewed through human-driven workflows. They are not the result of indiscriminate web scraping. The collection process emphasizes combat segments with visible player-boss interaction and usable action structure.

The current annotations are designed as research metadata rather than final polished captions. Some rows may still be marked as unverified while remaining useful for training, filtering, and prototyping.

Intended Uses

This dataset may be useful for:

• —Training interactive video generation models conditioned on action or language descriptions.
• —Training multi-entity world models under a single shared camera.
• —Training or evaluating game agents from video-action trajectories.
• —Studying temporal action grounding, combat event parsing, and video captioning.
• —Prototyping language-based control interfaces for game-world or embodied agents.

The dataset is not intended for commercial redistribution of game assets, impersonation of official game content, or use as a complete benchmark of Elden Ring gameplay.

Dataset Layout

margit/
  meta_data.jsonl
  videos/
    *.mp4
crucible_knight/
  meta_data.jsonl
  videos/
    *.mp4

Current contents:

• —margit/meta_data.jsonl: 729 rows.
• —margit/videos/: 729 MP4 clips referenced by the Margit metadata.
• —crucible_knight/meta_data.jsonl: 340 rows.
• —crucible_knight/videos/: 340 MP4 clips referenced by the Crucible Knight metadata.

These files are only a portion of the planned public data. They are representative of the annotation schema and training signal, but not exhaustive in boss coverage, action vocabulary, or world coverage.

The layout is intentionally scene-centric so new bosses, worlds, caption variants, and evaluation splits can be added without changing the existing structure.

Metadata Schema

Each JSONL row describes one video clip. The common fields are:

• —video: Source video filename. The corresponding file is stored under the scene's videos/ directory.
• —verified: Whether the annotation has been manually verified.
• —prompt.scene_clarity: Coarse scene quality label.
• —prompt.global_caption: Clip-level summary, visible objects, and time range.
• —prompt.participants: Per-entity timelines for the player and boss.
• —prompt.participants.*.timeline: Temporally ordered action segments with start/end times, action captions, and interaction flags.
• —statistics: Derived counts and interaction indicators.
• —context_used: Whether contextual information was used while preparing the annotation.

A simplified row has the following shape:

{
  "video": "20260117_180710_0.mp4",
  "verified": false,
  "prompt": {
    "scene_clarity": "Clear",
    "global_caption": {
      "short_caption": "Action sequence: ...",
      "long_caption": "The clip shows the character performing: ...",
      "visible_objects": ["Tarnished", "Boss", "Environment"],
      "start_time": 0.0,
      "end_time": 23.0
    },
    "participants": {
      "person_1": {
        "short_description": "Tarnished character.",
        "timeline": [
          {
            "start_time": 0.0,
            "end_time": 2.5,
            "short_caption": "Charge forward",
            "long_caption": "person_1 performs Charge forward.",
            "is_interaction": false,
            "interaction_with": []
          }
        ]
      },
      "boss_1": {
        "short_description": "Boss character.",
        "timeline": [
          {
            "start_time": 3.75,
            "end_time": 6.0,
            "short_caption": "Staff upswing",
            "long_caption": "boss_1 performs Staff upswing.",
            "is_interaction": true,
            "interaction_with": ["person_1"]
          }
        ]
      }
    }
  },
  "statistics": {
    "max_events_count": 11,
    "has_boss_interaction": true
  },
  "context_used": true
}

Loading

Individual configurations can be loaded with datasets:

from datasets import load_dataset

margit = load_dataset("zhush/incantation-elden-ring-scenes", "margit")
crucible_knight = load_dataset("zhush/incantation-elden-ring-scenes", "crucible_knight")

The video field stores the filename. To access the actual MP4 file, join it with the corresponding scene directory, for example margit/videos/{video_filename}.

Relationship to the Incantation Paper

Incantation: Natural Language as the Action Interface for Multi-Entity Video World Models introduces natural language as a per-frame, per-entity action interface for shared-camera video world models. It supports simultaneous multi-entity control and concept-level action transfer across entities and worlds.

This repository is the public preview subset associated with the paper. The paper-scale experiments use a larger dataset and training pipeline, including additional Elden Ring bosses and The King of Fighters data.

• —Paper: arXiv:2605.18601
• —Project page: matrixteam-ai.github.io/pages/Incanation
• —GitHub: zhushangwen/Incantation
• —Checkpoints: Not yet released

Limitations

• —This release is a preview subset and does not represent the full action, boss, world, or scale distribution used by the paper.
• —The current metadata is designed for research use and may contain imperfect captions or unverified rows.
• —Clip distributions are shaped by manual collection and filtering, not by a balanced benchmark protocol.
• —The dataset contains gameplay footage from a commercial game; downstream users should follow applicable research, legal, and platform constraints.

Citation

If you use this dataset, please cite the Incantation paper:

@article{zhu2026incantation,
  title={Incantation: Natural Language as the Action Interface for Multi-Entity Video World Models},
  author={Zhu, Shangwen and Peng, Qianyu and Pu, Zhao and Shu, Zhilei and Ke, Xiangrui and Xing, Zhaohu and Tong, Zizhao and Wang, Zeqing and Cui, Xinyu and Zheng, Zian and Wang, Huangji and Zhao, Jian and Jin, Yeying and Cheng, Fan and Feng, Ruili},
  journal={arXiv preprint arXiv:2605.18601},
  year={2026}
}

Legal Notice

Elden Ring is a trademark of FromSoftware, Inc. and Bandai Namco Entertainment Inc. This dataset is an academic research artifact and is not affiliated with, endorsed by, or sponsored by the game developers or publishers.