Datasetpublic

justachetan/flat-pack-bench

Flat-Pack Bench 🧩 Furniture assembly as a spatio-temporal stress test for large vision-language models. Flat-Pack Bench is a multiple-choice benchmark for evaluating fine-grained spatio-temporal understanding in real furniture assembly videos. Each question asks a model to reason about object parts, contact events, assembly order, final connectivity, or part identity across time. Project page: https://flat-pack-bench.github.io 🎯 Benchmark Tasks The benchmark… See the full description on the dataset page: https://huggingface.co/datasets/justachetan/flat-pack-bench.

sourceHugging Facemitupdated 4mo agoView on Hugging Face

0likes9.5kdownloads

Dataset Card

Flat-Pack Bench 🧩

Furniture assembly as a spatio-temporal stress test for large vision-language models.

Flat-Pack Bench is a multiple-choice benchmark for evaluating fine-grained spatio-temporal understanding in real furniture assembly videos. Each question asks a model to reason about object parts, contact events, assembly order, final connectivity, or part identity across time.

Project page: https://flat-pack-bench.github.io

🎯 Benchmark Tasks

The benchmark contains 602 multiple-choice questions organized into four families:

Family	Count	What it tests
Temporal Localization	103	Identify which connection happened first, last, next, or most recently relative to an assembly state.
Temporal Ordering	155	Recover the order in which parts or part-pairs become physically connected.
Mating	87	Determine whether two labeled parts are connected in the fully assembled furniture.
Tracking	257	Match labeled parts across two states, requiring object identity through occlusion and motion.

The Hugging Face Dataset Viewer is configured to display the benchmark question table from questions/questions.jsonl. Other JSON files in this repository are supporting annotations, mask files, and frame indexes rather than row-oriented question examples.

📁 Repository Structure

Path	Contents
`questions/`	Benchmark questions in JSONL form plus one YAML source file per question.
`videos/`	Raw, trimmed, and keyframe MP4 video variants.
`rgb-frames/`	Extracted RGB frames for videos, with source-frame-to-file mappings.
`segmentation-masks/`	Part-level masks encoded as compressed run-length encodings.
`furniture-annotations/`	Furniture part semantics, connectivity graphs, similar-part annotations, and OBJ part meshes.

Most media and annotation paths follow this key pattern:

text

<category>/<furniture_name>/<video_id>/

For example, the video ID d9sENgGhBk4 for Chair/ronninge can be found under matching paths in videos/, rgb-frames/, and, when available, segmentation-masks/.

🗂️ Main Schemas

Detailed schema notes live next to the relevant data:

—questions/README.md documents questions/questions.jsonl.
—questions/yamls/README.md documents questions/yamls/*.yaml.
—furniture-annotations/part-annotations/README.md documents per-furniture part graphs and semantics.
—rgb-frames/README.md documents frame image paths and frame_id_to_fn.json.
—segmentation-masks/README.md documents mask JSON files.
—videos/README.md documents video variants and path conventions.

🙏 Acknowledgments

Flat-Pack Bench is built around real furniture assembly videos and annotations from the IKEA Manuals at Work dataset. We would like to thank authors of IKEA Manuals at Work: 4D Grounding of Assembly Instructions on Internet Videos: Yunong Liu, Cristobal Eyzaguirre, Manling Li, Shubh Khanna, Juan Carlos Niebles, Vineeth Ravi, Saumitra Mishra, Weiyu Liu, and Jiajun Wu.

📚 Citation

If you use Flat-Pack Bench, please cite:

bibtex

@InProceedings{Chetan_2026_CVPR,
    author    = {Chetan, Aditya and Cai, Eric and Kushwaha, Peeyush and Kani, Bharath Raj Nagoor and Mall, Utkarsh and Wang, Qianqian and Snavely, Noah and Hariharan, Bharath},
    title     = {Flat-Pack Bench: Evaluating Spatio-Temporal Understanding in Large Vision-Language Models through Furniture Assembly},
    booktitle = {Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)},
    month     = {June},
    year      = {2026},
    pages     = {16624-16634}
}