Datasetpublic

neuripsedtracksub/ego-counterfactual-mistakes

Ego Counterfactual Mistakes (Ego-CoMist) Description This synthetic dataset contains mistake-intervention annotations for interactive cooking guidance. Each row contains video segment with instruction/feedback text pairs and their timestamps. Dataset Details Files: annotations.json Release statistics: Total rows: 25,087 Unique videos (dataset + video_id): 1,110 Rows by source dataset: CaptainCook4D: 4,969 Ego4D: 13,847 Ego-Exo4D: 6,271… See the full description on the dataset page: https://huggingface.co/datasets/neuripsedtracksub/ego-counterfactual-mistakes.

sourceHugging Faceotherupdated 5mo agoView on Hugging Face

0likes15downloads

Dataset Card

Ego Counterfactual Mistakes (Ego-CoMist)

Description

This synthetic dataset contains mistake-intervention annotations for interactive cooking guidance. Each row contains video segment with instruction/feedback text pairs and their timestamps.

Dataset Details

Files:

—annotations.json

Release statistics:

—Total rows: 25,087
—Unique videos (dataset + video_id): 1,110

Rows by source dataset:

—CaptainCook4D: 4,969
—Ego4D: 13,847
—Ego-Exo4D: 6,271

Dataset Collection Process

The dataset is generated using Qwen3-VL-32B-Instruct and Gemini-2.5-Pro.

Data Format

Each row in annotations.json has schema:

json

{
  "dataset": "Ego4D",
  "video_id": "39d087b0-afc2-47d8-ba91-b70dd8fab90e",
  "start_time": 1191.26881,
  "end_time": 1204.66377,
  "mistake_type": "preparation_error",
  "texts": [
    "Microwave the water until boiling.",
    "For this step, please boil the water in the microwave instead of on the stovetop."
  ],
  "text_timestamps": [1191.26881, 1196.26881],
  "text_types": ["Instruction", "Feedback"]
}

Usage

python

from datasets import load_dataset

ds = load_dataset("neuripsedtracksub/ego-counterfactual-mistakes", split="train")

row = ds[0]
source_dataset = row["dataset"]
video_id = row["video_id"]
texts = row["texts"]
text_types = row["text_types"]
text_timestamps = row["text_timestamps"]

Dataset License

This dataset is released for research purposes only.