wovenbytoyota-vai/InstVL

InstVL: A Large-Scale Instance-Aware Vision-Language Dataset

[image]

This is the official repository for the InstVL dataset, introduced in the paper InstAP: Instance-Aware Vision-Language Pre-Train for Spatial-Temporal Understanding.

InstVL is a large-scale dataset of images and videos designed to bridge the gap between holistic scene understanding and fine-grained, instance-level comprehension. Current vision-language pre-training (VLP) paradigms excel at global scene understanding but struggle with instance-level reasoning due to global-only supervision. InstVL addresses this by providing dual-granularity annotations that enable instance-aware pre-training.

InstVL provides two levels of detailed textual annotations:

• —Global Captions — A comprehensive description of the entire scene.
• —Instance Captions — Fine-grained descriptions grounded to specific object regions (in images) or spatial-temporal trajectories (in videos).

The dataset contains 2 million images and 50,000 videos, providing rich supervision for instance-centric pre-training and benchmarking.

💻 How to Load with 🤗 Datasets

You can load the dataset directly using the Hugging Face datasets library. You must specify which configuration (image or video) you want to load.

Loading the Image Dataset

from datasets import load_dataset

img_ds = load_dataset(
    "wovenbytoyota-vai/instvl",
    name="image",
    data_files={
        "train": "train/instvl_img_2m.jsonl",
        "test_1k": "test/instvl_img_1k.jsonl",
        "test_10k": "test/instvl_img_10k.jsonl",
        "test_zero_1k": "test/instvl_img_zero_1k.jsonl",
        "test_zero_10k": "test/instvl_img_zero_10k.jsonl",
    },
)

# Access a split
train_split = img_ds["train"]

print(train_split[0])

Loading the Video Dataset

from datasets import load_dataset

video_ds = load_dataset(
    "wovenbytoyota-vai/instvl",
    name="video",
    data_files={
        "train": "train/instvl_video_50k.jsonl",
        "test_1k": "test/instvl_video_1k.jsonl",
        "test_10k": "test/instvl_video_10k.jsonl",
    }
)
# Access a split
train_split = video_ds["train"]

# Print the first example
print(train_split[0])

📋 Dataset Structure

The dataset is organized into train and test splits, with data provided in the JSON Lines (.jsonl) format.

.
├── test
│   ├── instvl_img_10k.jsonl
│   ├── instvl_img_1k.jsonl
│   ├── instvl_img_zero_10k.jsonl
│   ├── instvl_img_zero_1k.jsonl
│   ├── instvl_video_10k.jsonl
│   └── instvl_video_1k.jsonl
└── train
    ├── instvl_img_2m.jsonl
    └── instvl_video_50k.jsonl

🖼️ Image Data Structure

Each line in the image .jsonl files represents a single image and its annotations.

Fields

instance_data Object

📹 Video Data Structure

Each line in the video .jsonl files represents a single video segment and its annotations.

Fields

instance_data Object

📊 Data Splits

The InstVL dataset is divided into several training and test splits to facilitate robust benchmarking.

Meaning of the "Zero" Splits

The img-zero splits are a key component for evaluation. These subsets are sourced entirely from COYO, whereas the main training images (and their corresponding test splits) are from LAION. This introduces a distribution shift that lets us confirm that a model's performance demonstrates true generalization capabilities and is not merely inherited from the training distribution.

🏆 Benchmark Results

The following tables show the performance of the InstAP model on the InstVL test sets and standard zero-shot benchmarks, compared to other state-of-the-art models. All results are from the original paper. UMT-L (InstVL; g) and UMT-L (InstVL; g+i) are baselines trained on the same full training corpus as InstAP, using only InstVL's global captions (g) or with all InstVL captions treated as global (g+i).

Instance-Level Retrieval Performance (R@1)

This task evaluates the model's ability to retrieve the correct fine-grained instance caption. We report T2V / V2T R@1.

Global Retrieval Performance (R@1)

This task evaluates the model's ability to retrieve the correct global caption for the entire scene. We report T2V / V2T R@1.

Zero-Shot Text-to-Video Retrieval (R@1 / R@5 / R@10)

This table evaluates generalization on standard zero-shot text-to-video retrieval benchmarks.

Visual Grounding Performance (IoU@{50, 70, 90}) on InstVL-1K

This task evaluates the model's ability to localize textual mentions to specific spatial-temporal regions.

⬇️ Downloading the Original Images & Videos

You can download the original images and videos from the following websites:

• —InstVL Image: LAION-400M via img2dataset
• —InstVL Image Zero: COYO-700M via img2dataset
• —InstVL Video: HD-VILA-100M on Hugging Face

🙏 Acknowledgements

This dataset is based on results obtained from a project, JPNP20017, subsidized by the New Energy and Industrial Technology Development Organization (NEDO).

📝 License

Refer to the license CC BY-NC-SA 4.0 for using our dataset.

📜 Citations

If you use this dataset in your research, please cite the original paper:

@misc{kumar2026instapinstanceawarevisionlanguagepretrain,
  title={InstAP: Instance-Aware Vision-Language Pre-Train for Spatial-Temporal Understanding},
  author={Ashutosh Kumar and Rajat Saini and Jingjing Pan and Mustafa Erdogan and Mingfang Zhang and Betty Le Dem and Norimasa Kobori and Quan Kong},
  year={2026},
  eprint={2604.08337},
  archivePrefix={arXiv},
  primaryClass={cs.CV},
  url={https://arxiv.org/abs/2604.08337},
}