Datasetpublic

qualcomm/qualcomm-interactive-cooking-dataset-counterfactual-mistakes

Qualcomm Interactive Cooking Dataset: Ego Counterfactual Mistakes Description This synthetic dataset contains mistake-intervention annotations for interactive cooking guidance. Each row contains video segment with instruction/feedback text pairs and their timestamps. Dataset Details Files: annotations.json Release statistics: Total rows: 25,087 Unique videos (dataset + video_id): 1,110 Rows by source dataset: CaptainCook4D: 4,969 Ego4D: 13,847… See the full description on the dataset page: https://huggingface.co/datasets/qualcomm/qualcomm-interactive-cooking-dataset-counterfactual-mistakes.

sourceHugging Faceotherupdated 5mo agoView on Hugging Face

1likes49downloads

Dataset Card

Qualcomm Interactive Cooking Dataset: Ego Counterfactual Mistakes

Description

This synthetic dataset contains mistake-intervention annotations for interactive cooking guidance. Each row contains video segment with instruction/feedback text pairs and their timestamps.

Dataset Details

Files:

—annotations.json

Release statistics:

—Total rows: 25,087
—Unique videos (dataset + video_id): 1,110

Rows by source dataset:

—CaptainCook4D: 4,969
—Ego4D: 13,847
—Ego-Exo4D: 6,271

Dataset Collection Process

The dataset is generated using Qwen3-VL-32B-Instruct and Gemini-2.5-Pro.

Data Format

Each row in annotations.json has schema:

json

{
  "dataset": "Ego4D",
  "video_id": "39d087b0-afc2-47d8-ba91-b70dd8fab90e",
  "start_time": 1191.26881,
  "end_time": 1204.66377,
  "mistake_type": "preparation_error",
  "texts": [
    "Microwave the water until boiling.",
    "For this step, please boil the water in the microwave instead of on the stovetop."
  ],
  "text_timestamps": [1191.26881, 1196.26881],
  "text_types": ["Instruction", "Feedback"]
}

Usage

python

from datasets import load_dataset

ds = load_dataset("qualcomm/qualcomm-interactive-cooking-dataset-counterfactual-mistakes", split="train")

row = ds[0]
source_dataset = row["dataset"]
video_id = row["video_id"]
texts = row["texts"]
text_types = row["text_types"]
text_timestamps = row["text_timestamps"]

Dataset License

This dataset is released for research purposes only. Use of the dataset is subject to the license terms of the Qualcomm Interactive Cooking Dataset. Please refer to the accompanying license documentation for full terms, conditions, and usage restrictions.

Dataset Citation Instructions

If you use this dataset, please cite:

bibtex

@inproceedings{interactivecooking,
 title      = {Can Multi-Modal LLMs Provide Live Step-by-Step Task Guidance?},
 author     = {Apratim Bhattacharyya and Bicheng Xu and Sanjay Haresh and Reza Pourreza and Litian Liu and Sunny Panchal and Leonid Sigal and Roland Memisevic},
 booktitle  = {NeurIPS},
 year       = {2025}
}