datasets
Training and evaluation data, with the modality, task and licence stated up front. Listed live from the Hugging Face Hub.
train_video_and_instruction
ShareGPTVideo Training Data
All dataset and models can be found at ShareGPTVideo.
Contents:
Train 300k video frames: contains video frames used for SFT and DPO model, which is a subset of total 900k.
ActivityNet 50k + vidal 150k + webvid 100k.
Train 600k video frames: contains the rest 600k frames, the total 900k frames are used for pre-training stage. If you just do finetuning using our video QA, you can just download the 300k above.
900k composition is 400k WebVid +… See the full description on the dataset page: https://huggingface.co/datasets/ShareGPTVideo/train_video_and_instruction.Temporal-Logic-Video-Dataset
Temporal Logic Video (TLV) Dataset
Temporal Logic Video (TLV) Dataset
Synthetic and real video dataset with temporal logic annotation
Explore the GitHub »
NSVS-TL Project Webpage
·
NSVS-TL Source Code
Overview
The Temporal Logic Video (TLV) Dataset addresses the scarcity of state-of-the-art video datasets for long-horizon, temporally extended activity and object detection. It comprises two main components:
Synthetic… See the full description on the dataset page: https://huggingface.co/datasets/minkyuchoi/Temporal-Logic-Video-Dataset.Videos-Dataset-For-LLMs-RAG-That-Require-Audio-Vidoes-And-Text
Dataset Overview
A collection of 27 domains (“topics”) and 3100 question-answer pair.
Each topic comes with average 117 QA pairs.Every QA entry comes with:
references: one or more source files the answer is extracted from
time with each reference comes the starting and ending time the answer is extracted from the reference
video_files: the video files where the answer can be found
(future) video title & description from metadata.csv
File structure
You-Are-Here!/… See the full description on the dataset page: https://huggingface.co/datasets/elmoghany/Videos-Dataset-For-LLMs-RAG-That-Require-Audio-Vidoes-And-Text.VideoHallucer
VideoHallucer
Paper: https://huggingface.co/papers/2406.16338
This work introduces VideoHallucer, the first comprehensive benchmark for hallucination detection in large video-language models (LVLMs). VideoHallucer categorizes hallucinations into two main types: intrinsic and extrinsic, offering further subcategories for detailed analysis, including object-relation, temporal, semantic detail, extrinsic factual, and extrinsic non-factual hallucinations. We adopt an adversarial binary… See the full description on the dataset page: https://huggingface.co/datasets/bigai-nlco/VideoHallucer.ego4d-videoEgoCOT is a large-scale embodied planning dataset, which selected egocentric videos from the Ego4D dataset and corresponding high-quality step-by-step language instructions, which are machine generated, then semantics-based filtered, and finally human-verified.
For mored details, please visit EgoCOT_Dataset.
If you find this dataset useful, please consider citing the paper,
@article{mu2024embodiedgpt,
title={Embodiedgpt: Vision-language pre-training via embodied chain of thought}… See the full description on the dataset page: https://huggingface.co/datasets/wofmanaf/ego4d-video.Audio-Video-Engineering-Agentic-Tasks-1M
Audio/Video Engineering Agentic Tasks (1M)
Abstract
A highly specialized dataset comprising 1,029,459 in-context troubleshooting prompts and execution commands built for the deepest levels of media production. Unlike standard datasets that simulate clean, theoretical instructions, this matrix captures the chaotic, highly-detailed, and conversational reality of professional audio engineers, composers, and video editors mid-session. It is engineered to train multimodal AI… See the full description on the dataset page: https://huggingface.co/datasets/yatin-superintelligence/Audio-Video-Engineering-Agentic-Tasks-1M.VideoThinkBench
[CVPR 2026] Thinking with Video: Video Generation as a Promising Multimodal Reasoning Paradigm
🎊 News
[2026.02] 🔥🔥Our work has been accepted by CVPR 2026! 🎉🎉🎉
[2025.11] Our paper "Thinking with Video: Video Generation as a Promising Multimodal Reasoning Paradigm" has been released on arXiv! 📄 [Paper] On HuggingFace, it has achieved "#1 Paper of the Day"!
[2025.11] 🔥We release "minitest" of our VideoThinkBench, including 500… See the full description on the dataset page: https://huggingface.co/datasets/OpenMOSS-Team/VideoThinkBench.VideoFeedback2📃Paper |
🌐Website |
💻Code |
🛢️Dataset (VideoFeedback2) |
🤗Model (VideoScore2) |
🤗Space (VideoScore2) |
🤗50K videos cache
Overview
VideoFeedback2 is a large-scale, human-annotated dataset designed for training and evaluating multi-dimensional video evaluator 🤗Model (VideoScore2). It contains 27,168 AI-generated videos paired with fine-grained human feedback scores and reasoning traces across three evaluation dimensions:
(1) Visual Quality; (2) Text Alignment; (3)… See the full description on the dataset page: https://huggingface.co/datasets/TIGER-Lab/VideoFeedback2.Multi-Source-Video-Captioning
Multi-source Video Captioning (MSVC) Dataset Card
Dataset details
Dataset type:
MSVC is a set of collected video captioning data. It is constructed to ensure a robust and thorough evaluation of Video-LLMs' video-captioning capabilities.
Dataset detail:
MSVC is introduced to address limitations in existing video caption benchmarks, MSVC samples a total of 1,500 videos with human-annotated captions from MSVD, MSRVTT, and VATEX, ensuring diverse scenarios and domains.… See the full description on the dataset page: https://huggingface.co/datasets/DAMO-NLP-SG/Multi-Source-Video-Captioning.train_raw_video
ShareGPTVideo Raw ActivityNet Videos for Train data
All dataset and models can be found at ShareGPTVideo.
Contents:
Due to our scene split, we provide our processed activityNet videos corresponding to test frames in
train video frames
the processing script is process_activitynet.py
minimal_video_pairs
Minimal Video Pairs
A shortcut-aware benchmark for spatio-temporal and intuitive physics video understanding (VideoQA) using minimally different video pairs.
Github
For legal reasons, we are unable to upload the videos directly to Huggingface. However, we provide scripts in this repository for downloading the videos in our github repository. Our benchmark is built on top of videos source from 9 domains:
Subset
Data sources
Human object interactions
PerceptionTest… See the full description on the dataset page: https://huggingface.co/datasets/facebook/minimal_video_pairs.Video-T3-QATextual Temporal Understanding Dataset
Temporal Reasoning Transfer from Text to Video, ICLR 2025
Project Page: https://video-t3.github.io/
In each json file, we provide LLaVA-style text QA samples, using the synthesization method described in our paper.
For example:
[
{
"from": "human",
"value": "Based on the following captions describing keyframes of a video, answer the next question.\n\nCaptions:\nThe image displays a circular emblem with a metallic appearance, conveying a… See the full description on the dataset page: https://huggingface.co/datasets/MMInstruction/Video-T3-QA.VideoHallucer
VideoHallucer
Paper: https://huggingface.co/papers/2406.16338
This work introduces VideoHallucer, the first comprehensive benchmark for hallucination detection in large video-language models (LVLMs). VideoHallucer categorizes hallucinations into two main types: intrinsic and extrinsic, offering further subcategories for detailed analysis, including object-relation, temporal, semantic detail, extrinsic factual, and extrinsic non-factual hallucinations. We adopt an adversarial binary… See the full description on the dataset page: https://huggingface.co/datasets/ov015/VideoHallucer.VideoFeedback2📃Paper |
🌐Website |
💻Code |
🛢️Dataset (VideoFeedback2) |
🤗Model (VideoScore2) |
🤗Space (VideoScore2) |
🤗50K videos cache
Overview
VideoFeedback2 is a large-scale, human-annotated dataset designed for training and evaluating multi-dimensional video evaluator 🤗Model (VideoScore2). It contains 27,168 AI-generated videos paired with fine-grained human feedback scores and reasoning traces across three evaluation dimensions:
(1) Visual Quality; (2) Text Alignment; (3)… See the full description on the dataset page: https://huggingface.co/datasets/tripwayz/VideoFeedback2.videobridge-ai-kb
🎬 VideoBridge AI — Knowledge Base (RAG)
The single knowledge base for the VideoBridge AI project — used by the Telegram bot (@VideoBrige_bot) and the site panel for RAG search (/ai/rag).
This is the only Hugging Face dataset created for this project (no duplicate models/spaces/databases).
🔄 Auto-update
A daily scraper (scraper/kb_scrape.py) refreshes this dataset with:
📚 Academic papers (arXiv): text-to-video, video generation, video diffusion, RAG
📰 AI/tech… See the full description on the dataset page: https://huggingface.co/datasets/sosa123454321/videobridge-ai-kb.magic-video-artifacts
MAGIC-Video — Preprocessing Artifacts
This dataset hosts the exact preprocessing artifacts used in the paper
"Bridging Modalities, Spanning Time: Structured Memory for Ultra-Long Agentic Video Reasoning"
(MAGIC-Video, arXiv:2605.08271).
Why release these?
The paper's preprocessing pipeline calls LLMs through OpenRouter (translation, OpenIE, semantic
consolidation, narrative chain distillation). Those calls cost money, take hours per subject,
and are non-deterministic — re-running… See the full description on the dataset page: https://huggingface.co/datasets/jiazhengli7/magic-video-artifacts.morse-500
MORSE-500 Benchmark
🔥 News
May 15, 2025: We release MORSE-500, 500 programmatically generated videos across six reasoning categories: abstract, mathematical, physical, planning, spatial, and temporal, to stress-test multimodal reasoning. Frontier models including OpenAI o3 and Gemini 2.5 Pro score lower than… See the full description on the dataset page: https://huggingface.co/datasets/video-reasoning/morse-500.VideoMind
🔍VideoMind: An Omni-Modal Video Dataset with Intent Grounding for Deep-Cognitive Video Understanding
Dataset Description
VideoMind is a large-scale video-centric multimodal dataset that can be used to learn powerful and transferable text-video representations
for video understanding tasks such as video question answering and video retrieval. The VideoMind dataset contains 105K(5K test for
only) video samples, each of which is accompanied by audio, as well as systematic… See the full description on the dataset page: https://huggingface.co/datasets/DixinChen/VideoMind.VideoChatOnline-IT
Overview
This dataset provides a comprehensive collection for Online Spatial-Temporal Understanding tasks, covering multiple domains including Dense Video Captioning, Video Grounding, Step Localization, Spatial-Temporal Action Localization, and Object Tracking.
Data Formation
Our pipeline begins with 96K high-quality samples curated from 5 tasks across 12 datasets. The conversion process enhances online spatiotemporal understanding through template transformation. We… See the full description on the dataset page: https://huggingface.co/datasets/MCG-NJU/VideoChatOnline-IT.test_video_and_instruction
ShareGPTVideo Testing Data
All dataset and models can be found at ShareGPTVideo.
Contents:
Test video frames: contains video frames used for testing
In-domain: WebVid, Vidal (Youtube shorts), ActivityNet
Out-of-domain: MSRVTT, MSVD, TGIF, SSV2
Testing pipeline refer to LLaVA-Hound-DPO test
Inference and results: result, replying to issue
DPO ranking results: dpo ranking
Set up:
git clone git@github.com:RifleZhang/LLaVA-Hound-DPO.git
source setup/setup_env.sh
source… See the full description on the dataset page: https://huggingface.co/datasets/ShareGPTVideo/test_video_and_instruction.VideoChat2-IT
Instruction Data
Annotations
A comprehensive dataset of 1.9M data annotations is available in JSON format. Due to the extensive size of the full data, we provide only JSON files here. For corresponding images and videos, please follow our instructions.
Source data
Image
For image datasets, we utilized M3IT, filtering out lower-quality data by:
Correcting typos: Most sentences with incorrect punctuation usage were rectified.
Rephrasing incorrect… See the full description on the dataset page: https://huggingface.co/datasets/OpenGVLab/VideoChat2-IT.cooking-videos-with-captionsDataset of cooking videos obtained from pexels.com. Captions have been generated using AI.
MT-Video-Bench
MT-Video-Bench: A Holistic Video Understanding Benchmark for Evaluating Multimodal LLMs in Multi-Turn Dialogues
✨ Introduction
Recent advances in multimodal large language models (MLLMs) have brought remarkable progress in video understanding.However, existing evaluation benchmarks remain limited to single-turn question answering, overlooking the complexity of multi-turn dialogues in real-world scenarios.
🎬 MT-Video-Bench fills this… See the full description on the dataset page: https://huggingface.co/datasets/NJU-LINK/MT-Video-Bench.multimodal-vision-language-video-models-2026
👁️ Multimodal Vision-Language & Video Foundation Models Dataset (2026 Edition)
A structured research dataset featuring 1,000 domain-verified research papers and code repositories focused on Multimodal Vision-Language Models (VLM), Video Foundation Models, Diffusion Transformers (DiT), Visual Grounding, and World Simulators.
Built with Universal Scientific Engine V15.1 Gold, providing 47 schema attributes with verified repository attribution, modality capability matrix, vision… See the full description on the dataset page: https://huggingface.co/datasets/beatsprom/multimodal-vision-language-video-models-2026.test_raw_video_data
ShareGPTVideo Raw Videos for Testing data
All dataset and models can be found at ShareGPTVideo.
Contents:
In case of need, this contains raw videos corresponding to test frames in
Test video frames
elv-halluc-videos
ELV-Halluc — videos + annotations
A self-contained mirror of the ELV-Halluc benchmark
(CVPR 2026), bundling the raw .mp4 files together with the annotations so the benchmark can be
run without sourcing videos separately.
Paper: arXiv:2508.21496
Original annotations: HLSv/ELV-Halluc (no videos)
Project page: https://elv-halluc.github.io/
This is an unofficial mirror. All credit for the benchmark goes to the original authors; please
cite their paper (below) rather than this… See the full description on the dataset page: https://huggingface.co/datasets/shuzhig/elv-halluc-videos.VideoSimpleQA
Video SimpleQA: Towards Factuality Evaluation in Large Video Language Models
📖 Overview
Video SimpleQA is the first comprehensive benchmark specifically designed for evaluating factual grounding capabilities in Large Video Language Models (LVLMs). Unlike existing video benchmarks that often involve subjective speculation or conflate factual grounding with reasoning skills, Video SimpleQA focuses exclusively on objective factuality evaluation through multi-hop… See the full description on the dataset page: https://huggingface.co/datasets/VideoSimpleQA/VideoSimpleQA.morse-500-view
MORSE-500 Benchmark
The viewing version of MORSE-500 Benchmark, which allows you to view the video in the webpage directly.
Dataset Structure
test/: Contains all MP4 video files
test/metadata.csv: Contains the dataset metadata, including video_path, query, ground_truth, question_text, and main_category
raw_videoVideoScienceBench
VideoScienceBench
A benchmark for evaluating video understanding and scientific reasoning in vision-language models. Each example pairs a textual description of an experiment (what is shown) with the correct scientific explanation (expected phenomenon).
Dataset Summary
Attribute
Value
Examples
160
Domains
Physics, Chemistry
Format
JSONL (prompt + expected phenomenon + vid)
Data Creation Pipeline
Each researcher selects two or more scientific… See the full description on the dataset page: https://huggingface.co/datasets/lmgame/VideoScienceBench.