datasets
Training and evaluation data, with the modality, task and licence stated up front. Listed live from the Hugging Face Hub.
IMAGE_UNDERSTANDINGA key question for understanding multimodal performance is analyzing the ability for a model to have basic
vs. detailed understanding of images. These capabilities are needed for models to be used in
real-world tasks, such as an assistant in the physical world. While there are many dataset for object detection
and recognition, there are few that test spatial reasoning and other more targeted task such as visual prompting.
The datasets that do exist are static and publicly available, thus… See the full description on the dataset page: https://huggingface.co/datasets/microsoft/IMAGE_UNDERSTANDING.kitchen-workspace-understanding-safe-manipulation
Kitchen Workspace Understanding & Safe Manipulation
Generated by datapack-import.ts
This dataset mirrors public data-pack render outputs from Physicl.
Each row represents one render view. The image column contains a stable URL to the primary render image uploaded under /data; image_path stores the relative repository path and data_commit_sha pins the Hugging Face dataset commit used by those URLs. Files are uploaded as downloaded unless optional PNG recompression is enabled by… See the full description on the dataset page: https://huggingface.co/datasets/physicl/kitchen-workspace-understanding-safe-manipulation.synthetic-code-understanding
SYNTHETIC-1
This is a subset of the task data used to construct SYNTHETIC-1. You can find the full collection here
multi-view-bathroom-scene-understanding-camera-relocalization
Multi-View Bathroom Scene Understanding & Camera Relocalization
Generated by datapack-import.ts
This dataset mirrors public data-pack render outputs from Physicl.
Each row represents one render view. The image column contains a stable URL to the primary render image uploaded under /data; image_path stores the relative repository path and data_commit_sha pins the Hugging Face dataset commit used by those URLs. Files are uploaded as downloaded unless optional PNG recompression is… See the full description on the dataset page: https://huggingface.co/datasets/physicl/multi-view-bathroom-scene-understanding-camera-relocalization.physical-ai-bench-understanding
Physical AI Bench - Understanding
PAI-Bench (Physical AI Bench) is a comprehensive benchmark designed to evaluate physical AI generation and understanding capabilities across various real-world scenarios. This particular dataset, PAI-Bench-U, focuses specifically on Video Understanding tasks, comprising 2,808 real-world cases with task-aligned metrics.
Paper: PAI-Bench: A Comprehensive Benchmark For Physical AI
Code: GitHub Repository
Citation
If you use Physical AI… See the full description on the dataset page: https://huggingface.co/datasets/shi-labs/physical-ai-bench-understanding.Thai-understanding
Thai-Understanding: Thai-SUP & XLSR-Thai
Overview
Thai-Understanding is an open-source repository that provides a solution for speech understanding in the Thai language. This repository includes:
Thai-SUP: The first open-source Thai speech understanding dataset, which includes over 1,000 hours of data across three tasks: Intent Classification (IC), Named Entity Recognition (NER), and Speech Rephrasing (SR).
XLSR-Thai: The first large-scale self-supervised learning (SSL)… See the full description on the dataset page: https://huggingface.co/datasets/mcshao/Thai-understanding.Agentic-Long-Context-Understanding-QA 📖 Agentic Long Context Understanding 📖
Self-Taught Agentic Long Context Understanding (Arxiv).
AgenticLU refines complex, long-context queries through self-clarifications and contextual grounding, enabling robust long-document understanding in a single pass.
Installation Requirements
This codebase is largely based on OpenRLHF and Helmet, kudos to them.
The requirements are the same
pip install openrlhf
pip install -r ./HELMET/requirements.txt… See the full description on the dataset page: https://huggingface.co/datasets/yzhuang/Agentic-Long-Context-Understanding-QA.CUAD_v1_Contract_Understanding_PDF
Dataset Card for Contract Understanding Atticus Dataset (CUAD) PDF
This dataset contains the PDFs and the full text of 509 commercial legal contracts from the original CUAD dataset. One of the original 510 contracts was removed due to being a scanned copy.
The extracted text was cleaned using clean-text.
The PDFs were encoded in base64 and added as the pdf_bytes_base64 feature.
You can easily and quickly load it:
dataset = load_dataset("dvgodoy/CUAD_v1_Contract_Understanding_PDF")… See the full description on the dataset page: https://huggingface.co/datasets/dvgodoy/CUAD_v1_Contract_Understanding_PDF.CUAD_v1_Contract_Understanding_clause_classification
Dataset Card for Contract Understanding Atticus Dataset (CUAD) Clause Classification
This dataset contains 13,155 labeled clauses extracted from 509 commercial legal contracts from the original CUAD dataset. One of the original 510 contracts was removed due to being a scanned copy.
The text was cleaned using clean-text.
You can easily and quickly load it:
dataset = load_dataset("dvgodoy/CUAD_v1_Contract_Understanding_clause_classification")
Dataset({
features: ['file_name'… See the full description on the dataset page: https://huggingface.co/datasets/dvgodoy/CUAD_v1_Contract_Understanding_clause_classification.physical-ai-bench-understanding-evalsroie_document_understanding
Dataset Card for "sroie_document_understanding"
Dataset Description
This dataset is an enriched version of SROIE 2019 dataset with additional labels for line descriptions and line totals for OCR and layout understanding.
Dataset Structure
DatasetDict({
train: Dataset({
features: ['image', 'ocr'],
num_rows: 652
})
})
Data Fields
{
'image': PIL Image object,
'ocr': [
# text box 1
{
'box':… See the full description on the dataset page: https://huggingface.co/datasets/arvindrajan92/sroie_document_understanding.Spatial_Understanding
Purpose
Spatial intelligence is a fundamental component of both Artificial General Intelligence (AGI) and Embodied AI, encompassing multiple cognitive levels — Perception, Understanding, and Extrapolation (referring to the work).
We construct a composite benchmark derived from several prior works and this testbed is designed to measure the Understanding level of spatial intelligence of AI models within the given visual cues.
Overview
The benchmark integrates three… See the full description on the dataset page: https://huggingface.co/datasets/LLDDSS/Spatial_Understanding.mckinsey_state_of_ai_doc_understanding
Mckinsey State Of Ai Doc Understanding
This dataset was generated using YourBench (v0.3.1), an open-source framework for generating domain-specific benchmarks from document collections.
Pipeline Steps
ingestion: Read raw source documents, convert them to normalized markdown and save for downstream steps
summarization: Perform hierarchical summarization: chunk-level LLM summaries followed by combine-stage reduction
chunking: Split texts into token-based single-hop and… See the full description on the dataset page: https://huggingface.co/datasets/yourbench/mckinsey_state_of_ai_doc_understanding.humor_understanding_combinedPPTBench-Understanding
PPTBench Understanding Dataset
A collection of PowerPoint slides with associated understanding tasks and metadata.
Dataset Structure
The dataset contains the following fields for each entry:
hash: Unique identifier for each slide
category: Category/topic of the slide
task: Understanding task associated with the slide
description: Description of the slide content
question: Question formulated for the slide
Usage
Loading the Dataset
You can load this… See the full description on the dataset page: https://huggingface.co/datasets/tyrionhuu/PPTBench-Understanding.finnlp2026-subtask3-polyfiqa
FinNLP 2026 Subtask 3 — PolyFiQA
Files:
train.parquet: 76 public labeled PolyFiQA examples supplied by the organizers.
test.parquet: 76 hidden-answer participant examples with stable row IDs.
SOURCE_DATASET_CARD.md: source PolyFiQA dataset documentation and citation.
task_id identifies a source financial-report document and repeats four times. The added id column is the unique submission key. Test answers remain in the private competition repository.
basic-understandingsiglip-doc-understanding-classifier
SigLIP Doc Understanding — Unanswerable Question Detection Dataset
A mixed answerable / unanswerable benchmark dataset built from DocVQA and MP-DocVQA, used to
train and evaluate the siglip-doc-understanding-classifier
unanswerable-question detector.
Each row pairs a document image with a question. Half of the questions are the original,
answerable DocVQA/MP-DocVQA questions; the other half are corrupted versions of those same
questions — modified so the document image no longer… See the full description on the dataset page: https://huggingface.co/datasets/giacolees/siglip-doc-understanding-classifier.OpenFinData-Intent-Understanding-Intruct
📊 Financial Intent Understanding Dataset
Dataset for
🎯 Financial-Intent-Understanding-with-LLMs 🤖
This dataset is specifically prepared for fine-tuning (SFT) language models on financial intent understanding tasks. The data follows the Alpaca instruction format and is hosted on HuggingFace.
📈 Dataset Overview
Dataset Name: OpenFinData-Intent-Understanding-Instruct
HuggingFace URL: klaylouis1932/OpenFinData-Intent-Understanding-Intruct… See the full description on the dataset page: https://huggingface.co/datasets/klaylouis1932/OpenFinData-Intent-Understanding-Intruct.IMAGE_UNDERSTANDINGA key question for understanding multimodal performance is analyzing the ability for a model to have basic
vs. detailed understanding of images. These capabilities are needed for models to be used in
real-world tasks, such as an assistant in the physical world. While there are many dataset for object detection
and recognition, there are few that test spatial reasoning and other more targeted task such as visual prompting.
The datasets that do exist are static and publicly available, thus… See the full description on the dataset page: https://huggingface.co/datasets/neelsj/IMAGE_UNDERSTANDING.finnlp2026-subtask1-greek-ner
FinNLP 2026 Subtask 1 — Greek Financial NER
Files:
train.parquet: public numeric-NER training data supplied by the organizers.
validation.parquet: public numeric-NER validation data.
test.parquet: 200 hidden-label participant examples with stable row IDs.
SOURCE_DATASET_CARD.md: source Plutus dataset documentation and citation.
The test set has 200 rows: 100 textual-NER and 100 numeric-NER prompts over the same 100 Greek financial passages. It contains only id, query, and… See the full description on the dataset page: https://huggingface.co/datasets/FinNLP-Multilingual-Understanding/finnlp2026-subtask1-greek-ner.finnlp2026-subtask2-japanese-icr
FinNLP 2026 Subtask 2 — Japanese Financial ICR
Files:
train.parquet: 253 public labeled examples.
test.parquet: 50 hidden-label participant examples.
SOURCE_DATASET_CARD.md: source JF-ICR dataset documentation.
The five labels are +2, +1, 0, -1, and -2. Test labels remain in the private competition repository.
humor_understanding_nytOpenFinData-Intent-Understanding
Financial Intent Understanding Dataset
Dataset Description
This dataset is an extension of Intent-Understanding Dataset from the OpenFinData project, specifically designed for small-scale Supervised Fine-Tuning (SFT).
Data Split Information
Train Set: Generated using Claude 3-5 Sonnet
Valid Set: Generated using Claude 3-5 Sonnet
Test Set: Original data from OpenFinData Release
Dataset Statistics
Train set size: {train_size} examples
Test set size:… See the full description on the dataset page: https://huggingface.co/datasets/klaylouis1932/OpenFinData-Intent-Understanding.Numerical_understandingquery_understanding_hardquery_understanding_extended
Query Understanding dataset extended
Filtered from Magpie set of datasets
Magpie-Phi3-Pro-300K-Filtered
Magpie-Align/Magpie-Gemma2-Pro-200K-Filtered
Magpie-Align/Magpie-Llama-3.1-Pro-300K-Filtered
Magpie-Align/Magpie-Qwen2-Pro-200K-English
argilla/magpie-ultra-v0.1
Magpie-Align/Magpie-Reasoning-150K
Filtering Steps
Difficulty : 'easy', 'medium', 'hard', 'very hard'
Instruct Reward: > -2
Task Categories
Math
Information seeking
Coding & Debugging
Reasoning
Data analysis… See the full description on the dataset page: https://huggingface.co/datasets/Isotonic/query_understanding_extended.query_understandings_v6query_understandings_v5merged-expert-understanding-libero-dataset