neuripsedtracksub/ego-mistake-corrections

Ego Mistake Corrections Benchmark (Ego-MC-Bench)

Description

This dataset contains cooking videos with timestamped instruction and feedback for task guidance.

Each row corresponds to one video and provides aligned lists of utterance text, utterance type, and timestamp.

Dataset Details

Release files:

• —annotations/annotations.json
• —videos/*.MP4

Release statistics:

• —Total videos: 40
• —Total released annotations: 1,597

Text type counts in annotations_release.json:

• —Instruction: 562
• —Success: 560
• —Feedback: 397
• —Instruction-Clarification: 26
• —User-Question: 22
• —User-Comment: 30

Dataset Collection Process

Annotations are timestamped and aligned with each video. They are manually annotated.

Data Format

Each row in annotations.json has schema:

{
  "video_id": "GX010021",
  "texts": [
    "Let's first preheat the oven to 350 degrees Fahrenheit.",
    "Great Job!"
  ],
  "text_types": [
    "Instruction",
    "Success"
  ],
  "text_timestamps": [
    4.418,
    15.982
  ]
}

Video asset for a sample video_id:

• —videos/<video_id>.MP4

Usage

from datasets import load_dataset

ds = load_dataset("neuripsedtracksub/ego-mistake-corrections", split="test")

row = ds[0]
video_id = row["video_id"]
texts = row["texts"]
text_types = row["text_types"]
text_timestamps = row["text_timestamps"]

# Video path convention:
# f"videos/{video_id}.MP4"

Dataset License

This dataset is released for research purposes only.