datasets
Training and evaluation data, with the modality, task and licence stated up front. Listed live from the Hugging Face Hub.
Nemotron-VLM-Dataset-v2
Nemotron-VLM-Dataset v2
Versions
Date
Commit
Changes
2025-11-05
head
Fix nights_cot dataset. Fix/filter broken <think> entries. Update fintabnet instructions. Update indexes.
2025-10-28
214051e
Initial Release
Dataset Description
Following up on Llama Nemotron VLM Dataset V1 with 3 million samples, we are releasing the Nemotron VLM Dataset V2 with almost three times as many high-quality samples.
This time, our focus was on three… See the full description on the dataset page: https://huggingface.co/datasets/nvidia/Nemotron-VLM-Dataset-v2.MSR-VTTClone from "friedrichor/MSR-VTT".
MSRVTT contains 10K video clips and 200K captions.
We adopt the standard 1K-A split protocol, which was introduced in JSFusion and has since become the de facto benchmark split in the Text-Video Retrieval field.
Train:
train_7k: 7,010 videos, 140,200 captions
train_9k: 9,000 videos, 180,000 captions
Test:
test_1k: 1,000 videos, 1,000 captions
🌟 Citation
@inproceedings{xu2016msrvtt,
title={Msr-vtt: A large video description dataset… See the full description on the dataset page: https://huggingface.co/datasets/VLM2Vec/MSR-VTT.DiDeMoClone from friedrichor/DiDeMo.
About
DiDeMo contains 10K long-form videos from Flickr. For each video, ~4 short sentences are annotated in temporal order. We follow the existing works to concatenate those short sentences and evaluate ‘paragraph-to-video’ retrieval on this benchmark.
We adopt the official split:
Train: 8,395 videos, 8,395 captions (concatenate from 33,005 short captions)
Val: 1,065 videos, 1,065 captions (concatenate from 4,290 short captions) (We don't have… See the full description on the dataset page: https://huggingface.co/datasets/VLM2Vec/DiDeMo.MSVDClone from "friedrichor/MSVD".
MSVD contains 1,970 videos, each of which is paired with ~40 captions.
We adopt the official split:
Train: 1,200 videos, 48,774 captions
Val: 100 videos, 4,290 captions
Test: 670 videos, 27,763 captions
🌟 Citation
@inproceedings{chen2011collecting,
title={Collecting highly parallel data for paraphrase evaluation},
author={Chen, David and Dolan, William B},
booktitle={Proceedings of the Annual Meeting of the Association for… See the full description on the dataset page: https://huggingface.co/datasets/VLM2Vec/MSVD.Llama-Nemotron-VLM-Dataset-v1
Llama-Nemotron-VLM-Dataset v1
Versions
Date
Commit
Changes
2025-08-11
bdb3899
Initial release
2025-08-18
5abc7df
Fixes bug (ocr_1 and ocr_3 images were swapped)
2025-08-19
ef85bef
Update instructions for ocr_9
2025-08-25
4e46f2b
Added example for Megatron Energon
2025-09-02
head
Update license headers
Quickstart
If you want to dive in right away and load some samples using Megatron Energon, check out this section below.
Data… See the full description on the dataset page: https://huggingface.co/datasets/nvidia/Llama-Nemotron-VLM-Dataset-v1.GMAI-VL-5.5M
GMAI-VL-5.5M Dataset
GMAI-VL-5.5M is a comprehensive, large-scale medical General Medical AI Vision-Language (GMAI-VL) dataset built specifically for training multimodal foundation models in the medical domain. It contains an extraordinary scale of high-quality instructions encompassing over 5.5 million multimodal question-answering pairs, carefully constructed based on hundreds of medical classification, segmentation, and detection datasets.
This repository… See the full description on the dataset page: https://huggingface.co/datasets/General-Medical-AI/GMAI-VL-5.5M.healthspend-dataMVBench
MVBench
Forked from https://huggingface.co/datasets/OpenGVLab/MVBench for reproducibility.
Important Update
[18/10/2024] Due to NTU RGB+D License, 320 videos from NTU RGB+D need to be downloaded manually. Please visit ROSE Lab to access the data. We also provide a list of the 320 videos used in MVBench for your reference.
We introduce a novel static-to-dynamic method for defining temporal-related tasks. By converting static tasks into dynamic ones, we facilitate… See the full description on the dataset page: https://huggingface.co/datasets/VLM2Vec/MVBench.sat-vl-sft-training-ready-v1
Dataset Summary
NuTonic/sat-bbox-metadata-sft-v1 is a metadata-first, procedural VLM SFT dataset built from an existing “sat-bbox” style dataset tree (Sentinel‑2 chips + per-tile JSON metadata sidecars, optionally paired Mapbox stills).
The goal is to create high-signal, production-shaped supervision for multimodal chat models:
Captioning for satellite chips
Grounding (bounding boxes in normalized coordinates) for land-cover regions
Class-focused captions and absence checks for… See the full description on the dataset page: https://huggingface.co/datasets/NuTonic/sat-vl-sft-training-ready-v1.Nemotron-VLM-Dataset-v2
Nemotron-VLM-Dataset v2
Versions
Date
Commit
Changes
2025-11-05
head
Fix nights_cot dataset. Fix/filter broken <think> entries. Update fintabnet instructions. Update indexes.
2025-10-28
214051e
Initial Release
Dataset Description
Following up on Llama Nemotron VLM Dataset V1 with 3 million samples, we are releasing the Nemotron VLM Dataset V2 with almost three times as many high-quality samples.
This time, our focus was on three main… See the full description on the dataset page: https://huggingface.co/datasets/introvoyz041/Nemotron-VLM-Dataset-v2.vlm-info-loss-results
VLM Grounding Evaluation Results
Grounding evaluation results for vision-language models on robotics manipulation datasets.
Part of the vlm-info-loss project studying
how VLM connectors transform visual representations.
Background
Our embedding-level analysis shows VLM connectors perform a compress-then-expand transformation:
they sharpen dominant-object representations while compressing secondary-object category identity.
All tested models converge to ~83%… See the full description on the dataset page: https://huggingface.co/datasets/MicroAGI-Labs/vlm-info-loss-results.MedReason
MedReason: Eliciting Factual Medical Reasoning Steps in LLMs via Knowledge Graphs
📃 Paper ｜🤗 MedReason-8B | 📚 MedReason Data
✨ Latest News
[05/27/2025] 🎉 MedReason wins 3rd prize🏆 in the Huggingface Reasoning Datasets Competition!
⚡Introduction
MedReason is a large-scale high-quality medical reasoning dataset designed to enable faithful and explainable medical problem-solving in large language models (LLMs).
We utilize a structured medical knowledge… See the full description on the dataset page: https://huggingface.co/datasets/UCSC-VLAA/MedReason.Vlaserpixie
Pixie Dataset
This dataset contains data and pre-trained models for the paper Pixie: Fast and Generalizable Supervised Learning of 3D Physics from Pixels.
Project Page: https://pixie-3d.github.io/
Code: https://github.com/vlongle/pixie
Contents
checkpoints_continuous_mse/: Continuous material property prediction model checkpoints
checkpoints_discrete/: Discrete material classification model checkpoints
real_scene_data/: Real scene data for evaluation… See the full description on the dataset page: https://huggingface.co/datasets/vlongle/pixie.W2-VLA-CoT
World-to-Wrist: Offline CoT Labels
This dataset contains frame-aligned offline chain-of-thought annotations used
to train W²-VLA policies on LIBERO, RoboTwin, and four real-world manipulation
tasks. Matching LeRobot action data is available in W2-VLA-Training-Data.
Dataset Structure
W2-VLA-CoT/
├── libero/
│ ├── libero_10_no_noops_1.0.0_lerobot/
│ ├── libero_goal_no_noops_1.0.0_lerobot/
│ ├── libero_object_no_noops_1.0.0_lerobot/
│ └──… See the full description on the dataset page: https://huggingface.co/datasets/yuuu94/W2-VLA-CoT.XLRS-Bench-lite_VLM
🐙GitHub
Information or evaluatation on this dataset can be found in this repo: https://github.com/AI9Stars/XLRS-Bench
📜Dataset License
Annotations of this dataset is released under a Creative Commons Attribution-NonCommercial 4.0 International License. For images from:
DOTARGB images from Google Earth and CycloMedia (for academic use only; commercial use is prohibited, and Google Earth terms of use apply).
ITCVDLicensed under CC-BY-NC-SA-4.0.
MiniFrance… See the full description on the dataset page: https://huggingface.co/datasets/initiacms/XLRS-Bench-lite_VLM.nornikel-metallurgy-vl-dataset
Nornikel Metallurgy VL Dataset (SFT / DPO / GRPO)
Датасет для дообучения мультимодальной модели Qwen3-VL по схеме
SFT → DPO → GRPO в предметной области металлургии, горного дела и
обогащения полезных ископаемых. Построен из корпуса технических документов
(PDF-книги/сборники, DOCX-отчёты, PPTX-презентации, XLSX-таблицы) и
изображений (схемы, диаграммы, таблицы).
Конфигурации (config_name)
config
train
validation
назначение
sft
111 351
12 372… See the full description on the dataset page: https://huggingface.co/datasets/brics-edtech/nornikel-metallurgy-vl-dataset.HMDB51UCF101SmthSmthV2Robot-VLA-R1BreakfastVL-Interaction-EN
VL-Interaction-EN — 纯英文高质量训练子集(源视频 + 标注)
从复现的 1.42M 样本中筛选的 179,979 条纯英文高质量子集(chat 90K / narration 50K / event_grounding 40K),含源视频与逐秒标注。筛选方法见 SUBSET_REPORT_en.md(annotations_en.tar 内)。
内容
annotations_en.tar — 逐秒训练标注:annotations_en/<task>__<source>.jsonl(每行一个样本:messages/images/video_name/video_path)+ 筛选清单 subset_en.jsonl + 报告
data/<源>__NNN.tar — 对应源视频,≤10G 独立分片,解压得 videos_pool/<源>/<视频>
用法
tar xf annotations_en.tar
for t in data/*.tar; do tar xf… See the full description on the dataset page: https://huggingface.co/datasets/momo321654/VL-Interaction-EN.VLM-3R-DATA
VLM-3R Training Data
Training QA data for VLM-3R: vsibench_train/ (VSI-Bench-style tasks) and
vstibench_train/ (VSTI-Bench tasks over ScanNet train split).
Erratum (2026-07-13): corrected camera-position ground truth
A bug in the QA generation pipeline (reported by
Jacob Yeung, CMU) extracted
the camera center from camera-to-world poses using -R.T @ t instead of
pose[:3, 3]. Answers in five vstibench_train files depended on the camera's
world position and have… See the full description on the dataset page: https://huggingface.co/datasets/Journey9ni/VLM-3R-DATA.sat-vl-sft-postprocessed-merged-v1
Dataset Summary
NuTonic/sat-bbox-metadata-sft-v1 is a metadata-first, procedural VLM SFT dataset built from an existing “sat-bbox” style dataset tree (Sentinel‑2 chips + per-tile JSON metadata sidecars, optionally paired Mapbox stills).
The goal is to create high-signal, production-shaped supervision for multimodal chat models:
Captioning for satellite chips
Grounding (bounding boxes in normalized coordinates) for land-cover regions
Class-focused captions and absence checks for… See the full description on the dataset page: https://huggingface.co/datasets/NuTonic/sat-vl-sft-postprocessed-merged-v1.reward-projection-goal-generalisation-vlmClinSeek-Bench
ClinSeek-Bench
ClinSeek-Bench is the evaluation suite introduced in
ClinSeekAgent: Automating Multimodal Evidence Seeking for Agentic Clinical
Reasoning. It evaluates clinical reasoning
under two paired settings with the same task definitions and answer labels:
Curated Input: the model answers from the evidence package provided by
the source benchmark.
Automated Evidence-Seeking: the curated context is removed, and the model
must retrieve evidence from raw clinical data using… See the full description on the dataset page: https://huggingface.co/datasets/UCSC-VLAA/ClinSeek-Bench.VL-DocIR
Abstract
VL-DocIR is a page-level benchmark for vision-based long-document retrieval built from 29,641 documents rendered into 388,548 page images from Wikipedia, arXiv, PubMed, and SEC proxy statements. The benchmark contains 271,760 questions over 23 domains and six query types, covering single-page, multi-page, and cross-document evidence configurations. Questions are grounded to rendered pages and HTML element identifiers, then filtered with a cleaning pipeline that targets… See the full description on the dataset page: https://huggingface.co/datasets/anonymous-8421/VL-DocIR.OmniEarth-Bench_MCQ_VLMvla-reasoning