datasets
Training and evaluation data, with the modality, task and licence stated up front. Listed live from the Hugging Face Hub.
multi_task_multi_modal_knowledge_retrieval_benchmark_M2KR
PreFLMR M2KR Dataset Card
Dataset details
Dataset type:
M2KR is a benchmark dataset for multimodal knowledge retrieval. It contains a collection of tasks and datasets for training and evaluating multimodal knowledge retrieval models.
We pre-process the datasets into a uniform format and write several task-specific prompting instructions for each dataset. The details of the instruction can be found in the paper. The M2KR benchmark contains three types of tasks:… See the full description on the dataset page: https://huggingface.co/datasets/BByrneLab/multi_task_multi_modal_knowledge_retrieval_benchmark_M2KR.multi_task_multi_modal_knowledge_retrieval_benchmark_M2KR_CN
PreFLMR M2KR Dataset Card
Dataset details
Dataset type:
M2KR is a benchmark dataset for multimodal knowledge retrieval. It contains a collection of tasks and datasets for training and evaluating multimodal knowledge retrieval models.
We pre-process the datasets into a uniform format and write several task-specific prompting instructions for each dataset. The details of the instruction can be found in the paper. The M2KR benchmark contains three types of tasks:… See the full description on the dataset page: https://huggingface.co/datasets/BByrneLab/multi_task_multi_modal_knowledge_retrieval_benchmark_M2KR_CN.SAE_activations_modal_sentencesbimanual-table-cleanup-cross-embodiment-rich-modality-sample
Cross-Embodiment Bimanual Table Cleanup — Rich-Modality 10-Episode Inspection Sample
10 full-modality cross-embodiment bimanual table-cleanup episodes: 5 Franka Panda + 5 WidowXAI, 21,267 frames, 6 RGB views per robot, task-camera depth and segmentation, native robot state/action, end-effector trajectories, 6-DoF object poses, and QA annotations.
✅ Use it / ❌ Skip it
Use it for
Inspecting loaders, schemas, camera coverage, depth, segmentation, object poses… See the full description on the dataset page: https://huggingface.co/datasets/ExylosAi/bimanual-table-cleanup-cross-embodiment-rich-modality-sample.NEXUS-temporal_hierarchical_multi-modal
NEXUS: Neural Evolution for eXtensible Universal Semantics Dataset
(Temporal Multimodal Slices)
This dataset is a multi-modal, hierarchical, temporal representation derived from HuggingFaceFV/finevideo. It is designed for streaming training where the primary unit is a 10 ms "slice" that aggregates upward into moments (100 ms), seconds (1 s), experiences (10 s), and minutes (60 s).
It is meant to represent an extensible stream of "experience" as there are… See the full description on the dataset page: https://huggingface.co/datasets/Ardea/NEXUS-temporal_hierarchical_multi-modal.coffee-brew-cross-embodiment-rich-modality-sample
Cross-Embodiment Coffee Brew — Rich-Modality 10-Episode Inspection Sample
10 full-modality cross-embodiment coffee-brew episodes: 5 single-arm Franka Panda + 5 single-arm WidowXAI, 14,401 frames, 5 RGB views per robot, task-camera depth and segmentation, native robot state/action, end-effector trajectories, 6-DoF object poses, and QA annotations.
✅ Use it / ❌ Skip it
Use it for
Inspecting loaders, schemas, camera coverage, depth, segmentation, object poses… See the full description on the dataset page: https://huggingface.co/datasets/ExylosAi/coffee-brew-cross-embodiment-rich-modality-sample.multi-modal-derived-brain-network
PPMI Connectivity Graphs — HF Staging (Derivatives)
This dataset ships ready-to-use functional brain connectivity graphs derived from the PPMI cohort in a BIDS-ish derivatives layout. For each subject and parcellation, we include:
ROI time-series (*_desc-timeseries_parc-<name>.mat)
Pearson correlation connectivity matrix (*_desc-correlation_matrix_parc-<name>.mat)
JSON sidecars with summary fields (nodes, measure, symmetric/weighted flags)
Contents
data/… See the full description on the dataset page: https://huggingface.co/datasets/pakkinlau/multi-modal-derived-brain-network.autoinference-agentic-mix-v1
Autoinference Agentic Mix v1
This is a prompt set for the online_agentic serving benchmark. That profile stands
in for long-horizon agent traffic: a large context that grows turn over turn, with
short structured outputs at each step. The usual way to run it uses
generated-shared-prefix, which builds a synthetic shared prefix out of random tokens.
This dataset uses real agent trajectories instead, so the prefix reuse, the context
growth, and the token mix all match what an agent… See the full description on the dataset page: https://huggingface.co/datasets/modal-labs/autoinference-agentic-mix-v1.object-sorting-cross-embodiment-rich-modality-sample
Cross-Embodiment Object Sorting — Rich-Modality 20-Episode Inspection Sample
20 episodes total: 10 Franka Panda + 10 WidowXAI. A compact cross-embodiment inspection release for picking up an instructed object and placing it into an instructed target box. Each robot keeps its native LeRobot v2.1 state/action schema in a separate Viewer config. Five synchronized RGB views, metric depth and instance segmentation for every camera, robot state/action, end-effector trajectories… See the full description on the dataset page: https://huggingface.co/datasets/ExylosAi/object-sorting-cross-embodiment-rich-modality-sample.multi-modal-peg-in-square-hole-testThis dataset was created using LeRobot.
Dataset Structure
meta/info.json:
{
"codebase_version": "v2.1",
"robot_type": "ur5",
"total_episodes": 51,
"total_frames": 8074,
"total_tasks": 1,
"total_videos": 153,
"total_chunks": 1,
"chunks_size": 1000,
"fps": 10,
"splits": {
"train": "0:51"
},
"data_path": "data/chunk-{episode_chunk:03d}/episode_{episode_index:06d}.parquet",
"video_path":… See the full description on the dataset page: https://huggingface.co/datasets/hainh22/multi-modal-peg-in-square-hole-test.autoinference-agentic-mix-v2
Autoinference Agentic Mix v2
300 real SWE-agent trajectories from
TIGER-Lab/SWE-Next-SFT-Trajectories,
expanded into one request per assistant turn. Each row carries the conversation
up to that turn and the model generates the turn. Replaying a trajectory in
turn_index order re-sends a growing prefix, which is how an agent loop
actually hits a prefix cache.
What changed from v1
v1 kept only requests with at least 34k prefix tokens. That cut trajectories
down to… See the full description on the dataset page: https://huggingface.co/datasets/modal-labs/autoinference-agentic-mix-v2.shared-emergence-icl-modalities-128
Shared-emergence ICL replication at T=128
This dataset contains the complete raw result archive for the paper
“Many Next-Token Predictors are In-Context Learners.”
The campaign evaluates a fixed suite of 100 program-synthesis tasks using 128
sampled prompts per task, for every clean and deranged shot cell described by
the paper:
21 run keys;
281 experiment cells;
12,800 predictions per cell;
3,596,800 predictions in total.
The archive expands to a top-level results_128/… See the full description on the dataset page: https://huggingface.co/datasets/N8Programs/shared-emergence-icl-modalities-128.multi-modal-peg-in-square-hole-image-aloneThis dataset was created using LeRobot.
Dataset Structure
meta/info.json:
{
"codebase_version": "v2.1",
"robot_type": "ur5",
"total_episodes": 11,
"total_frames": 1892,
"total_tasks": 1,
"total_videos": 33,
"total_chunks": 1,
"chunks_size": 1000,
"fps": 10,
"splits": {
"train": "0:11"
},
"data_path": "data/chunk-{episode_chunk:03d}/episode_{episode_index:06d}.parquet",
"video_path":… See the full description on the dataset page: https://huggingface.co/datasets/hainh22/multi-modal-peg-in-square-hole-image-alone.full-modality-data
Full Modality Dataset Statistics
Video Statistics
Total Videos: 28,472
Total Duration: 1422.33 hours
Average Duration: 179.84 seconds
Median Duration: 160.08 seconds
Duration Range: 10.04s - 1780.03s
QA Statistics
Total Questions: 1,444,526
Average Questions per Video: 50.7
Questions per Video Range: 14 - 450
Question Type Distribution
OE: 1,444,526 (100.0%)
Question Category Distribution
temporal: 96,873 (6.7%)
causal: 96,873… See the full description on the dataset page: https://huggingface.co/datasets/lmms-lab/full-modality-data.FW_EDU_SUBSET_500k_docs
FineWeb-Edu Subset
This dataset contains 483,606 documents sampled from the FineWeb-Edu dataset.
The dataset is used throughout various tutorials on modalities.
For licensing, see their conditions.
mod-alg2
Dataset Card for "mod-alg2"
More Information needed
multi_modality_testThis dataset was created using LeRobot.
Dataset Structure
meta/info.json:
{
"codebase_version": "v2.0",
"robot_type": "koch",
"total_episodes": 6,
"total_frames": 2543,
"total_tasks": 1,
"total_videos": 18,
"total_chunks": 1,
"chunks_size": 1000,
"fps": 30,
"splits": {
"train": "0:6"
},
"data_path": "data/chunk-{episode_chunk:03d}/episode_{episode_index:06d}.parquet",
"video_path":… See the full description on the dataset page: https://huggingface.co/datasets/seeingrain/multi_modality_test.multi-modal-peg-in-square-holeThis dataset was created using LeRobot.
Dataset Structure
meta/info.json:
{
"codebase_version": "v2.1",
"robot_type": "ur5",
"total_episodes": 201,
"total_frames": 28616,
"total_tasks": 1,
"total_videos": 603,
"total_chunks": 1,
"chunks_size": 1000,
"fps": 10,
"splits": {
"train": "0:201"
},
"data_path": "data/chunk-{episode_chunk:03d}/episode_{episode_index:06d}.parquet",
"video_path":… See the full description on the dataset page: https://huggingface.co/datasets/hainh22/multi-modal-peg-in-square-hole.rollout_act_so101_earplug_cup_v3_modal_20260702_155144This dataset was created using LeRobot.
Dataset Structure
meta/info.json:
{
"codebase_version": "v3.0",
"fps": 30,
"features": {
"action": {
"dtype": "float32",
"names": [
"shoulder_pan.pos",
"shoulder_lift.pos",
"elbow_flex.pos",
"wrist_flex.pos",
"wrist_roll.pos",
"gripper.pos"
],
"shape": [
6… See the full description on the dataset page: https://huggingface.co/datasets/AlexTranscend/rollout_act_so101_earplug_cup_v3_modal_20260702_155144.so101_teleop_mat_to_tray_with_modalitiesThis dataset was created using LeRobot.
Dataset Structure
meta/info.json:
{
"codebase_version": "v3.0",
"robot_type": "so101_follower",
"total_episodes": 10,
"total_frames": 2020,
"total_tasks": 1,
"chunks_size": 1000,
"data_files_size_in_mb": 100,
"video_files_size_in_mb": 200,
"fps": 30,
"splits": {
"train": "0:10"
},
"data_path": "data/chunk-{chunk_index:03d}/file-{file_index:03d}.parquet",
"video_path":… See the full description on the dataset page: https://huggingface.co/datasets/liorbenhorin-nv/so101_teleop_mat_to_tray_with_modalities.europe-owid-modal-age-at-death-in-adults
Modal Age At Death In Adults | Europe (Our World in Data)
🇪🇺 238 observations · 5 Europe countries · 1970–2022 · Repackaged by Electric Sheep Europe
TL;DR
This dataset contains 238 observations of Modal Age At Death In Adults data across 5 Europe countries, spanning 1970–2022.
About the source
Source: Our World in Data
Publisher: Our World in Data
License: cc-by-4.0
Topic: Modal Age At Death In Adults
Geographic coverage
5… See the full description on the dataset page: https://huggingface.co/datasets/electricsheepeurope/europe-owid-modal-age-at-death-in-adults.MULTI_VALUE_qqp_present_modals
Dataset Card for "MULTI_VALUE_qqp_present_modals"
More Information needed
rollout_act_so101_earplug_cup_v3_modal_20260702_155430This dataset was created using LeRobot.
Dataset Structure
meta/info.json:
{
"codebase_version": "v3.0",
"fps": 30,
"features": {
"action": {
"dtype": "float32",
"names": [
"shoulder_pan.pos",
"shoulder_lift.pos",
"elbow_flex.pos",
"wrist_flex.pos",
"wrist_roll.pos",
"gripper.pos"
],
"shape": [
6… See the full description on the dataset page: https://huggingface.co/datasets/AlexTranscend/rollout_act_so101_earplug_cup_v3_modal_20260702_155430.rollout_act_so101_earplug_cup_v3_modal_20260702_155854This dataset was created using LeRobot.
Dataset Structure
meta/info.json:
{
"codebase_version": "v3.0",
"fps": 30,
"features": {
"action": {
"dtype": "float32",
"names": [
"shoulder_pan.pos",
"shoulder_lift.pos",
"elbow_flex.pos",
"wrist_flex.pos",
"wrist_roll.pos",
"gripper.pos"
],
"shape": [
6… See the full description on the dataset page: https://huggingface.co/datasets/AlexTranscend/rollout_act_so101_earplug_cup_v3_modal_20260702_155854.new_screw_ticks_yolo_modalityThis dataset was created using LeRobot.
Dataset Structure
meta/info.json:
{
"codebase_version": "v2.1",
"robot_type": "mcx",
"total_episodes": 2,
"total_frames": 1748,
"total_tasks": 1,
"total_videos": 4,
"total_chunks": 1,
"chunks_size": 1000,
"fps": 30,
"splits": {
"train": "0:2"
},
"data_path": "data/chunk-{episode_chunk:03d}/episode_{episode_index:06d}.parquet",
"video_path":… See the full description on the dataset page: https://huggingface.co/datasets/mortiferr/new_screw_ticks_yolo_modality.MULTI_VALUE_mnli_double_modals
Dataset Card for "MULTI_VALUE_mnli_double_modals"
More Information needed
MULTI_VALUE_mnli_present_modals
Dataset Card for "MULTI_VALUE_mnli_present_modals"
More Information needed
MULTI_VALUE_mrpc_present_modals
Dataset Card for "MULTI_VALUE_mrpc_present_modals"
More Information needed
MULTI_VALUE_rte_present_modals
Dataset Card for "MULTI_VALUE_rte_present_modals"
More Information needed
MULTI_VALUE_stsb_double_modals
Dataset Card for "MULTI_VALUE_stsb_double_modals"
More Information needed