datasets
Training and evaluation data, with the modality, task and licence stated up front. Listed live from the Hugging Face Hub.
LLaVA-OneVision-1.5-Mid-Training-85M
🚀 LLaVA-One-Vision-1.5-Mid-Training-85M Dataset is being uploaded 🚀
Upload Status
All Completed: ImageNet-21k、LAIONCN、DataComp-1B、Zero250M、COYO700M、SA-1B、MINT、Obelics
📜 Cite
If you find LLaVA-One-Vision-1.5-Mid-Training-85M useful in your research, please consider to cite the following related papers:
@misc{an2025llavaonevision15fullyopenframework,
title={LLaVA-OneVision-1.5: Fully Open Framework for Democratized Multimodal Training}… See the full description on the dataset page: https://huggingface.co/datasets/mvp-lab/LLaVA-OneVision-1.5-Mid-Training-85M.LLaVA-OneVision-2-Data
LLaVA-OneVision-2-Data
Training data for the LLaVA-OneVision-2 multimodal model family. The release contains large-scale video data at several duration ranges, video captions and source mappings, and spatial-reasoning data used for mid-training.
At a Glance
The dataset is split across two Hugging Face repositories because of its size:
Repository
What it contains
Part 1 (this repository)
~60-second video shards, captions for all duration ranges… See the full description on the dataset page: https://huggingface.co/datasets/mvp-lab/LLaVA-OneVision-2-Data.LLaVA-OneVision-1.5-Instruct-Data
LLaVA-OneVision-1.5 Instruction Data
Paper | Code
📌 Introduction
This dataset, LLaVA-OneVision-1.5-Instruct, was collected and integrated during the development of LLaVA-OneVision-1.5. LLaVA-OneVision-1.5 is a novel family of Large Multimodal Models (LMMs) that achieve state-of-the-art performance with significantly reduced computational and financial costs. This meticulously curated 22M instruction dataset (LLaVA-OneVision-1.5-Instruct) is part of a… See the full description on the dataset page: https://huggingface.co/datasets/mvp-lab/LLaVA-OneVision-1.5-Instruct-Data.LLaVA-OneVision-2-Data-Part2LLaVA-Video-178K
Dataset Card for LLaVA-Video-178K
Uses
This dataset is used for the training of the LLaVA-Video model. We only allow the use of this dataset for academic research and education purpose. For OpenAI GPT-4 generated data, we recommend the users to check the OpenAI Usage Policy.
Data Sources
For the training of LLaVA-Video, we utilized video-language data from five primary sources:
LLaVA-Video-178K: This dataset includes 178,510 caption entries, 960,792 open-ended… See the full description on the dataset page: https://huggingface.co/datasets/lmms-lab/LLaVA-Video-178K.LLaVA-OneVision-Data
Dataset Card for LLaVA-OneVision
[2024-09-01]: Uploaded VisualWebInstruct(filtered), it's used in OneVision Stage
almost all subsets are uploaded with HF's required format and you can use the recommended interface to download them and follow our code below to convert them.
the subset of ureader_kg and ureader_qa are uploaded with the processed jsons and tar.gz of image folders.
You may directly download them from the following url.… See the full description on the dataset page: https://huggingface.co/datasets/lmms-lab/LLaVA-OneVision-Data.LLaVA-ReCap-CC12MLLaVA-OneVision-1.5-Instruct-Data-qwen-formatllavallava-bench-in-the-wild
Large-scale Multi-modality Models Evaluation Suite
Accelerating the development of large-scale multi-modality models (LMMs) with lmms-eval
🏠 Homepage | 📚 Documentation | 🤗 Huggingface Datasets
This Dataset
This is a formatted version of LLaVA-Bench(wild) that is used in LLaVA. It is used in our lmms-eval pipeline to allow for one-click evaluations of large multi-modality models.
@misc{liu2023improvedllava,
author={Liu, Haotian and Li, Chunyuan… See the full description on the dataset page: https://huggingface.co/datasets/lmms-lab-encoder/llava-bench-in-the-wild.LLaVA-OneVision-Data-ru
LLaVA-OneVision-Data-ru
Translated lmms-lab/LLaVA-OneVision-Data dataset into Russian language using Google translate.
Almost all datasets have been translated, except for the following:
["tallyqa(cauldron,llava_format)", "clevr(cauldron,llava_format)", "VisualWebInstruct(filtered)", "figureqa(cauldron,llava_format)", "magpie_pro(l3_80b_mt)", "magpie_pro(qwen2_72b_st)", "rendered_text(cauldron)", "ureader_ie"]
Usage
import datasets
data =… See the full description on the dataset page: https://huggingface.co/datasets/d0rj/LLaVA-OneVision-Data-ru.LLaVA-Instruct-150K
LLaVA Visual Instruct 150K Dataset Card
Dataset details
Dataset type:
LLaVA Visual Instruct 150K is a set of GPT-generated multimodal instruction-following data.
It is constructed for visual instruction tuning and for building large multimodal towards GPT-4 vision/language capability.
Dataset date:
LLaVA Visual Instruct 150K was collected in April 2023, by prompting GPT-4-0314 API.
Paper or resources for more information:
https://llava-vl.github.io/
License:… See the full description on the dataset page: https://huggingface.co/datasets/liuhaotian/LLaVA-Instruct-150K.LLaVA-OneVision-1.5-Mid-Training-Webdataset-Quick-Start-3MLLaVA-Pretrain
LLaVA Visual Instruct Pretrain Dataset Card
Dataset details
Dataset type:
LLaVA Visual Instruct Pretrain LCS-558K is a subset of LAION/CC/SBU dataset, filtered with a more balanced concept coverage distribution.
Captions are also associated with BLIP synthetic caption for reference.
It is constructed for the pretraining stage for feature alignment in visual instruction tuning.
We aim to build large multimodal towards GPT-4 vision/language capability.
Dataset date:
LLaVA… See the full description on the dataset page: https://huggingface.co/datasets/liuhaotian/LLaVA-Pretrain.LLaVA-NeXT-Data
Dataset Card for LLaVA-NeXT
We provide the whole details of LLaVA-NeXT Dataset. In this dataset, we include the data that was used in the instruction tuning stage for LLaVA-NeXT and LLaVA-NeXT(stronger).
Aug 30, 2024: We update the dataset with raw format (de-compress it for json file and images with structured folder), you can directly download them if you are familiar with LLaVA data format.
Dataset Sources
Compared to the instruction data mixture for LLaVA-1.5… See the full description on the dataset page: https://huggingface.co/datasets/lmms-lab/LLaVA-NeXT-Data.llava-video-178k-siglip-tokens-ftov-new
LLaVA-Video-178K SigLIP Token Cache (LLaVA-OV fine-tuned vision tower)
Derived data (vision-encoder features of video frames), not a
redistribution of the source videos. Source:
lmms-lab/LLaVA-Video-178K -- its card
restricts use to academic research and education, and its annotations come
from GPT-4-class models (see the OpenAI usage policy).
Complete: 85000 clips.
Subset
Folders: 0_30_s_academic_v0_1, 0_30_s_youtube_v0_1, 30_60_s_academic_v0_1… See the full description on the dataset page: https://huggingface.co/datasets/Ahmed-Nasri/llava-video-178k-siglip-tokens-ftov-new.llava_v1_5_mix665k
LLaVA v1.5 Mix 665K Dataset
This dataset contains 665,298 multimodal instruction-following samples used for fine-tuning the LLaVA v1.5 model.
Dataset Structure
id: Unique identifier for the sample
model: Model name (if applicable)
conversations: JSON string containing conversation turns in original format
image: List of PIL Image objects (embedded in parquet)
image_path: List of strings containing original relative paths to images
Load the Dataset
from… See the full description on the dataset page: https://huggingface.co/datasets/Icey444/llava_v1_5_mix665k.LLaVA-NeXT-Interleave-Bench
LLaVA-Interleave Bench Dataset Card
Dataset details
Dataset type:
LLaVA-Interleave Bench is a comprehensive set of multi-image datasets that are collected from public datasets or generated by the GPT-4V API.
It is constructed for evaluating the interleaved multi-image reaoning capbilities of LMMs.
Dataset date:
LLaVA-Interleave Bench was collected in April 2024, and released in June 2024.
Paper or resources for more information:
Blog:… See the full description on the dataset page: https://huggingface.co/datasets/lmms-lab-encoder/LLaVA-NeXT-Interleave-Bench.llava-instruct-mix
LLaVA Instruct Mix
Added OCR and Chart QA dataset into this for more text extraction questions
Agri-LLaVA_Agricultural_Pests_And_Diseases_Feature_Alignment_Dataset
Agri-LLaVA
Agri-LLaVA is a large multimodal instruction dataset for agriculture, pairing crop/leaf images with multi-turn diagnostic conversations about plant diseases, pests, and nutrient deficiencies. It is compiled from 16 public source datasets (see the license table below).
This dataset has been converted to Parquet format with image bytes embedded directly, standardized to the HF image_text_to_text format with a single conversational messages schema.
This dataset is… See the full description on the dataset page: https://huggingface.co/datasets/Project-AgML/Agri-LLaVA_Agricultural_Pests_And_Diseases_Feature_Alignment_Dataset.LLaVA-ReCap-CC3Mllava-finetuneLLaVA-CoT-100k
Dataset Card for LLaVA-CoT
The LLaVA-CoT-100k dataset is introduced in the paper LLaVA-CoT: Let Vision Language Models Reason Step-by-Step. This dataset is designed to enable Vision-Language Models (VLMs) to perform autonomous multistage reasoning, integrating samples from various visual question-answering sources with structured reasoning annotations. It aims to address the challenges VLMs face in systematic and structured reasoning for complex visual question-answering tasks.… See the full description on the dataset page: https://huggingface.co/datasets/Xkev/LLaVA-CoT-100k.llava-instruct-mix-vsfttheblackcat102/llava-instruct-mix reformated for VSFT with TRL's SFT Trainer.
See https://github.com/huggingface/trl/blob/main/examples/scripts/vsft_llava.py.
llava-665k
LLaVA-v1.5 Mix665K — Arrow (images embedded)
The LLaVA-v1.5 visual instruction-tuning mixture (llava_v1_5_mix665k) converted to a 🤗 datasets
Arrow dataset with image bytes embedded. 665,298 examples
(624,610 image–text + 40,688 text-only).
⚠️ This repo is a raw save_to_disk Arrow snapshot. The dataset viewer and
load_dataset() do not work here — load it with load_from_disk as shown below.
Loading
from huggingface_hub import snapshot_download
from datasets… See the full description on the dataset page: https://huggingface.co/datasets/Ethlake/llava-665k.LLAVA-LibMoE
Download Instructions
This repository provides the full LLAVA-LibMoE dataset, including LLaVA-665K and OneVision-1M2 image sources, organized into the following required directory tree:
libmoe/
└── data/
├── image_onevision/
├── coco/
│ └── train2017/
├── gqa/
│ └── images/
├── ocr_vqa/
│ └── images/
├── textvqa/
│ └── train_images/
└── vg/
├── VG_100K/
└──… See the full description on the dataset page: https://huggingface.co/datasets/DavidNguyen/LLAVA-LibMoE.LLaVA-UHD-v2-SFT-Data
LLaVA-UHD v2 SFT Data
Datasets used in the SFT stage of LLaVA-UHD v2(https://github.com/thunlp/LLaVA-UHD)
Usage
Please refer to the following script to decompress the dataset.
cat LLaVA-UHD-v2-SFT-Data.tar.part* > LLaVA-UHD-v2-SFT-Data.tar
tar -xvf ./LLaVA-UHD-v2-SFT-Data.tar
Detailed composition
total 858k-mixed:
LLaVA 158K
ShareGPT 40K
VQAv2 83K
GQA 72K
OKVQA 9K
OCRVQA 80K
DocVQA 15K
ChartQA 20K
A-OKVQA 66K
DVQA 20K
TextCaps 22K
ShareGPT4V 55K
AI2D 3K… See the full description on the dataset page: https://huggingface.co/datasets/YipengZhang/LLaVA-UHD-v2-SFT-Data.llava-en-zh-300kThis dataset is composed by
150k examples of English Visual Instruction Data from LLaVA.
150k examples of English Visual Instruction Data from openbmb.
You can use it in LLaMA Factory by specifying --dataset llava_150k_en,llava_150k_zh.
pred_llava_next_10kllava_trajectories