datasets
Training and evaluation data, with the modality, task and licence stated up front. Listed live from the Hugging Face Hub.
GMAI-VL-5.5M
GMAI-VL-5.5M Dataset
GMAI-VL-5.5M is a comprehensive, large-scale medical General Medical AI Vision-Language (GMAI-VL) dataset built specifically for training multimodal foundation models in the medical domain. It contains an extraordinary scale of high-quality instructions encompassing over 5.5 million multimodal question-answering pairs, carefully constructed based on hundreds of medical classification, segmentation, and detection datasets.
This repository… See the full description on the dataset page: https://huggingface.co/datasets/General-Medical-AI/GMAI-VL-5.5M.MVBench
MVBench
Forked from https://huggingface.co/datasets/OpenGVLab/MVBench for reproducibility.
Important Update
[18/10/2024] Due to NTU RGB+D License, 320 videos from NTU RGB+D need to be downloaded manually. Please visit ROSE Lab to access the data. We also provide a list of the 320 videos used in MVBench for your reference.
We introduce a novel static-to-dynamic method for defining temporal-related tasks. By converting static tasks into dynamic ones, we facilitate… See the full description on the dataset page: https://huggingface.co/datasets/VLM2Vec/MVBench.sat-vl-sft-training-ready-v1
Dataset Summary
NuTonic/sat-bbox-metadata-sft-v1 is a metadata-first, procedural VLM SFT dataset built from an existing “sat-bbox” style dataset tree (Sentinel‑2 chips + per-tile JSON metadata sidecars, optionally paired Mapbox stills).
The goal is to create high-signal, production-shaped supervision for multimodal chat models:
Captioning for satellite chips
Grounding (bounding boxes in normalized coordinates) for land-cover regions
Class-focused captions and absence checks for… See the full description on the dataset page: https://huggingface.co/datasets/NuTonic/sat-vl-sft-training-ready-v1.vlm-info-loss-results
VLM Grounding Evaluation Results
Grounding evaluation results for vision-language models on robotics manipulation datasets.
Part of the vlm-info-loss project studying
how VLM connectors transform visual representations.
Background
Our embedding-level analysis shows VLM connectors perform a compress-then-expand transformation:
they sharpen dominant-object representations while compressing secondary-object category identity.
All tested models converge to ~83%… See the full description on the dataset page: https://huggingface.co/datasets/MicroAGI-Labs/vlm-info-loss-results.pixie
Pixie Dataset
This dataset contains data and pre-trained models for the paper Pixie: Fast and Generalizable Supervised Learning of 3D Physics from Pixels.
Project Page: https://pixie-3d.github.io/
Code: https://github.com/vlongle/pixie
Contents
checkpoints_continuous_mse/: Continuous material property prediction model checkpoints
checkpoints_discrete/: Discrete material classification model checkpoints
real_scene_data/: Real scene data for evaluation… See the full description on the dataset page: https://huggingface.co/datasets/vlongle/pixie.nornikel-metallurgy-vl-dataset
Nornikel Metallurgy VL Dataset (SFT / DPO / GRPO)
Датасет для дообучения мультимодальной модели Qwen3-VL по схеме
SFT → DPO → GRPO в предметной области металлургии, горного дела и
обогащения полезных ископаемых. Построен из корпуса технических документов
(PDF-книги/сборники, DOCX-отчёты, PPTX-презентации, XLSX-таблицы) и
изображений (схемы, диаграммы, таблицы).
Конфигурации (config_name)
config
train
validation
назначение
sft
111 351
12 372… See the full description on the dataset page: https://huggingface.co/datasets/brics-edtech/nornikel-metallurgy-vl-dataset.sat-vl-sft-postprocessed-merged-v1
Dataset Summary
NuTonic/sat-bbox-metadata-sft-v1 is a metadata-first, procedural VLM SFT dataset built from an existing “sat-bbox” style dataset tree (Sentinel‑2 chips + per-tile JSON metadata sidecars, optionally paired Mapbox stills).
The goal is to create high-signal, production-shaped supervision for multimodal chat models:
Captioning for satellite chips
Grounding (bounding boxes in normalized coordinates) for land-cover regions
Class-focused captions and absence checks for… See the full description on the dataset page: https://huggingface.co/datasets/NuTonic/sat-vl-sft-postprocessed-merged-v1.VL-DocIR
Abstract
VL-DocIR is a page-level benchmark for vision-based long-document retrieval built from 29,641 documents rendered into 388,548 page images from Wikipedia, arXiv, PubMed, and SEC proxy statements. The benchmark contains 271,760 questions over 23 domains and six query types, covering single-page, multi-page, and cross-document evidence configurations. Questions are grounded to rendered pages and HTML element identifiers, then filtered with a cleaning pipeline that targets… See the full description on the dataset page: https://huggingface.co/datasets/anonymous-8421/VL-DocIR.VCR-Bench
VCR-Bench ( A Comprehensive Evaluation Framework for Video Chain-of-Thought Reasoning)
🌐 Homepage | 🤗 Dataset | 🤗 Paper | 📖 arXiv | GitHub
Dataset Details
As shown in the figure below, current video benchmarks often lack comprehensive annotations of CoT steps, focusing only on the accuracy of final answers during model evaluation while neglecting the quality of the reasoning process. This evaluation approach makes it difficult to comprehensively evaluate model’s… See the full description on the dataset page: https://huggingface.co/datasets/VLM-Reasoning/VCR-Bench.pcbslm-static-v2-unsloth-vlm
PCBSLM static-v2 Unsloth VLM
Portable multimodal Unsloth dataset for PCB layout/document-grounded training.
The JSONL splits use Unsloth/Gemma-style chat messages:
{
"messages": [
{"role": "user", "content": [
{"type": "image", "image": "assets/raw_docs/.../images/page.png"},
{"type": "text", "text": "instruction..."}
]},
{"role": "assistant", "content": [
{"type": "text", "text": "{...json answer...}"}
]}
]
}
Files… See the full description on the dataset page: https://huggingface.co/datasets/henry1477/pcbslm-static-v2-unsloth-vlm.PALL-VLM-data
PALL-VLM-data — Dental Vision-Language Dataset
The training dataset for Harisundar/PALL-VLM,
a dental vision-language model. It contains 32,884 records over 52,461 images,
formatted as image+text conversations for LLaVA-style instruction tuning.
Curated by: Harisundar R
Used by: Harisundar/PALL-VLM · PALL on GitHub
Language: English
Layout
vlm_train/
├── images/ # 52,461 dental images
├── train.jsonl # 29,667 records
├── val.jsonl… See the full description on the dataset page: https://huggingface.co/datasets/Harisundar/PALL-VLM-data.openvivqa-formating-vlm
OpenViVQA Formatting Dataset for VLM
A Vietnamese multimodal instruction-format dataset for training Vision Language Models (VLMs) on Visual Question Answering (VQA) tasks.
This dataset reformats OpenViVQA-style samples into conversational instruction-tuning format compatible with modern VLM training pipelines such as:
Qwen2-VL
LLaVA
InternVL
Phi-3 Vision
Idefics
SmolVLM
Dataset Structure
Each sample contains:
image: input image
conversations: multi-turn… See the full description on the dataset page: https://huggingface.co/datasets/Nhanvi282/openvivqa-formating-vlm.robotrace-vla-robustness-traces
RoboTrace Evidence Bundle
This dataset repository contains the public evidence bundle for RoboTrace, a low-cost deployment-stress evaluation scaffold for robot-learning and VLA-style inference pipelines.
The current release evaluates lerobot/pusht and includes reports, metrics, plots, summaries, and release manifests from a complete staged run.
What this bundle is for
Use this repository to inspect evidence from RoboTrace:
action-trace stability metrics
visual… See the full description on the dataset page: https://huggingface.co/datasets/i-am-shaurya05/robotrace-vla-robustness-traces.VL3-Syn7M
The re-caption dataset used in VideoLLaMA 3: Frontier Multimodal Foundation Models for Video Understanding
If you like our project, please give us a star ⭐ on Github for the latest update.
🌟 Introduction
This dataset is the re-captioned data we used during the training of VideoLLaMA3. It consists of 7 million diverse, high-quality images, each accompanied by a short caption and a detailed caption.
The images in this dataset originate from COYO-700M, MS-COCO 2017… See the full description on the dataset page: https://huggingface.co/datasets/DAMO-NLP-SG/VL3-Syn7M.aitf-dfk3-vlm-dataset-jsonlTBStar-VLM-R2VLM_semantics_SLO_benchmark
VLM Semantics SLO Benchmark
VLM Semantics SLO is a Slovenian multimodal benchmark for studying cultural and semiotic reasoning in vision-language models. It goes beyond object recognition by asking models to interpret visual hierarchy, spatial relations, colour and mood, composition, cultural symbols, metaphor, denotation and connotation, intertextuality, communicative intent, and relevance to Slovenia.
The released JSON contains 4,950 image-level records. Every record has ten… See the full description on the dataset page: https://huggingface.co/datasets/maticmatusek/VLM_semantics_SLO_benchmark.chart-vllm-ver1qwen3-vl-failure-cases
Qwen3-VL-2B-Instruct Failure Analysis Dataset
📊 Dataset Overview
This dataset contains 10 diverse failure cases identified while testing the Qwen3-VL-2B-Instruct vision-language model. Each example captures a specific type of error, providing valuable insights for targeted fine-tuning.
Failure Category
Count
Examples
Time Reading
2
Clock misreading (11:55 vs 10:10; 3:35 vs 10:35)
Counting
2
Remote buttons (3 vs 0); Strawberries (4 vs 1)
Negation… See the full description on the dataset page: https://huggingface.co/datasets/TasneemSelim/qwen3-vl-failure-cases.ViLReward-73KProcess Reward Data for ViLBench: A Suite for Vision-Language Process Reward Modeling
Paper | Project Page
There are 73K vision-language process reward data sourcing from five training sets.
ogiri-bokete-unsloth-vlm
Japanese Bokete Ogiri — Unsloth VLM format
YANS-official/ogiri-bokete を、UnslothのVision SFTで扱える会話形式に変換した非公開用データセットです。
各JSONLレコードは「1画像 + 1回答」です。
{
"messages": [
{"role": "user", "content": [
{"type": "image", "image": "images/124469.jpg"},
{"type": "text", "text": "この画像のお題に対して、面白い一言を1つ返してください。"}
]},
{"role": "assistant", "content": [
{"type": "text", "text": "..."}
]}
]
}
Files
train.jsonl: 1,678 records / 630 prompts… See the full description on the dataset page: https://huggingface.co/datasets/beezza/ogiri-bokete-unsloth-vlm.vlm_tsr_test_1
vlm_tsr_test_1
vlm_tsr_test의 1/4 파트. scene 그룹 50000~50004 포함.
전체 테스트셋은 4개 레포로 나뉘어 있습니다:
vlm_tsr_test_1
vlm_tsr_test_2
vlm_tsr_test_3
vlm_tsr_test_4
코드 및 전체 파이프라인: Lim-Sung-Jun/vlm_training_template
구조
각 샘플은 3개 파일 세트로 구성됩니다:
test/source/T01_C01/{id}.jpg # 테이블 이미지
test/source/T01_C01/{id}.json # 정답 HTML + 메타데이터
test/label/T01_C01/{id}.html # 렌더링용 GT HTML
평가 메트릭
메트릭
설명
TEDS
Tree-Edit Distance 기반 구조 유사도 (0~1)
TEDS-Structure
텍스트 제외 구조만… See the full description on the dataset page: https://huggingface.co/datasets/sungjun12/vlm_tsr_test_1.vla-evaluation-v3whatsup_vlmsvlfeedback_lowavgvlfeedback_highavgvlfeedback_fullmkn_vlm_datasetVL-DocIR-RepresentativeSubset
Representative Subset Creation
This dataset represents a representative subset of VL-DocIR dataset (https://huggingface.co/datasets/anonymous-8421/VL-DocIR).
Subset creation method:
Random sampling of 5 queries for the combination of each data source and evidence structure type (we ensure that no query is taken more than once).
This results in 120 queries.
Collection of all documents that are referenced by query evidence.
Abstract
VL-DocIR is a page-level benchmark… See the full description on the dataset page: https://huggingface.co/datasets/anonymous-8421/VL-DocIR-RepresentativeSubset.vlfeedback_highvar