datasets
Training and evaluation data, with the modality, task and licence stated up front. Listed live from the Hugging Face Hub.
computer-use-data-psai
Computer Use Dataset - PSAI
A large-scale, multimodal dataset of human-computer interactions for training and evaluating AI agents.
🔗 Access Dataset: https://huggingface.co/datasets/anaisleila/computer-use-data-psai
📊 Dataset Overview
This dataset contains 3,167 completed tasks of human-computer interactions captured with video, screenshots, DOM snapshots, and detailed interaction events. Created by Paradigm Shift AI for advancing computer use AI agent research.… See the full description on the dataset page: https://huggingface.co/datasets/anaisleila/computer-use-data-psai.vidore_v3_computer_scienceViDoRe V3 : Computer Science
This dataset, Computer Science, is a corpus of textbooks from the openstacks website, intended for long-document understanding tasks. It is one of the 10 corpora comprising the ViDoRe v3 Benchmark.
About ViDoRe v3
ViDoRe V3 is our latest benchmark for RAG evaluation on visually-rich documents from real-world applications. It features 10 datasets with, in total, 26,000 pages and 3099 queries, translated into 6 languages. Each query comes with… See the full description on the dataset page: https://huggingface.co/datasets/vidore/vidore_v3_computer_science.minecraft-vla-stage1
Minecraft VLA Stage 1: Action Pretraining Data
Vision-Language-Action training data for Minecraft, processed from OpenAI's VPT contractor dataset.
Dataset Description
This dataset contains frame-action pairs from Minecraft gameplay, designed for training VLA models following the Lumine methodology.
Source
Original: OpenAI VPT Contractor Data (7.x subset)
Videos: 17,886 videos (330 hours of early-game gameplay)
Task: "Play Minecraft" with focus on first 30… See the full description on the dataset page: https://huggingface.co/datasets/TESS-Computer/minecraft-vla-stage1.computer-use
Computer Use Trajectories
Successful computer-use agent trajectories collected on OSWorld tasks.
Dataset Details
Rows: 160 (one per task trajectory)
Steps: 1,378 total across all trajectories (avg ~8.6 steps/task)
Agent: Gemini 3 Flash Preview with linearized accessibility-tree grounding
Score filter: Only trajectories with score = 1.0 (fully successful)
Domains
Domain
Tasks
Description
chrome
21
Web browsing tasks in Google Chrome
gimp
15
Image… See the full description on the dataset page: https://huggingface.co/datasets/markov-ai/computer-use.minecraft-vla-stage2
Minecraft VLA Stage 2: Instruction-Following Data
Stage 2 of the TESS-Minecraft Vision-Language-Action training pipeline.
Overview
This dataset adds task instructions to the Stage 1 visuomotor data, enabling instruction-following training.
Data Format
Field
Type
Description
id
string
Unique sample ID
video_id
string
Source video name
frame_idx
int
Frame index within video
instruction
string
Task instruction (empty for continuation frames)… See the full description on the dataset page: https://huggingface.co/datasets/TESS-Computer/minecraft-vla-stage2.vidore_v3_computer_science_mteb_format
Vidore3ComputerScienceRetrieval
An MTEB dataset
Massive Text Embedding Benchmark
Retrieve associated pages according to questions.
Task category
t2i
Domains
Academic
Reference
https://huggingface.co/blog/QuentinJG/introducing-vidore-v3
Source datasets:
vidore/vidore_v3_computer_science
How to evaluate on this task
You can evaluate an embedding model on this dataset using the following code:
import mteb
task =… See the full description on the dataset page: https://huggingface.co/datasets/vidore/vidore_v3_computer_science_mteb_format.tess-agentnet
TESS AgentNet Dataset
Computer use trajectories for training Vision-Language-Action models.
Features
image: Screenshot (PIL Image)
instruction: Task description
action_type: 0=MOUSE, 1=KEYBOARD
mouse_x, mouse_y: Normalized coordinates [0,1]
click_type: 0-8 (NO_CLICK, LEFT_CLICK, etc.)
keyboard_text: Text with special tokens
os_type: ubuntu, windows_macos
episode_id, step_idx: Episode structure
Click Types
Index
Type
Description
0
NO_CLICK… See the full description on the dataset page: https://huggingface.co/datasets/TESS-Computer/tess-agentnet.atari-vla-stage1-5hz
TESS-Atari Stage 1 (5Hz)
Human gameplay demonstrations from Atari games, formatted for Vision-Language-Action (VLA) model training.
Overview
Metric
Value
Source
Atari-HEAD
Games
11 (overlapping with DIAMOND benchmark)
Samples
~4M
Action Rate
5 Hz (1 action per observation)
Format
Lumine-style action tokens
Games Included
Alien, Asterix, BankHeist, Breakout, DemonAttack, Freeway, Frostbite, Hero, MsPacman, RoadRunner, Seaquest… See the full description on the dataset page: https://huggingface.co/datasets/TESS-Computer/atari-vla-stage1-5hz.atari-vla-stage1-15hz
TESS-Atari Stage 1 (15Hz)
Human gameplay demonstrations from Atari games with action chunking, formatted for Vision-Language-Action (VLA) model training.
Overview
Metric
Value
Source
Atari-HEAD
Games
11 (overlapping with DIAMOND benchmark)
Samples
~1.3M
Observation Rate
5 Hz
Action Rate
15 Hz (3 actions per observation)
Format
Lumine-style action tokens
Why Action Chunking?
VLA models run at ~5 Hz inference speed, but Atari runs at… See the full description on the dataset page: https://huggingface.co/datasets/TESS-Computer/atari-vla-stage1-15hz.synthetic-computers-at-scale
Synthetic Computers
Paper: Synthetic Computers at Scale for Long-Horizon Productivity Simulation (arXiv:2604.28181)
A dataset of 98 synthetic computer environments designed for research on
computer-use agents, long-horizon planning, and persona-grounded reasoning.
Each row describes a single fictional user's computer — including the user's
persona, professional context, monthly objectives, collaborators, project
portfolio, filesystem policy, full file listing, and a graph of file… See the full description on the dataset page: https://huggingface.co/datasets/microsoft/synthetic-computers-at-scale.csgo-vla-stage1-5hz
CS:GO VLA Stage 1 Dataset (5Hz Chunked)
Vision-Language-Action dataset for Counter-Strike: Global Offensive with action chunking, converted from the TeaPearce CS:GO dataset.
Overview
Frame rate: 5Hz (every 3rd frame)
Action chunking: 3 actions per sample (~200ms coverage)
Total samples: ~1.8M chunks
Split: train / test following Diamond split
Map: Dust2 deathmatch
Action Format
<|action_start|> m1_x m1_y [keys1] ; m2_x m2_y [keys2] ; m3_x m3_y [keys3]… See the full description on the dataset page: https://huggingface.co/datasets/TESS-Computer/csgo-vla-stage1-5hz.mmlu-computer_security
Dataset Card for "mmlu-computer_security"
More Information needed
tess-atari-5hz-384IndustryCorpus2_computer_programming_code
IndustryCorpus2: Programming
This repository contains the IndustryCorpus2: Programming domain subset of BAAI/IndustryCorpus2.
Refer to the parent dataset card for data construction, intended use, limitations,
and licensing details.
Citation
If you use this dataset in your work, please cite IndustryCorpus2:
@misc{shi2024industrycorpus2,
title = {IndustryCorpus2},
author = {Xiaofeng Shi and Lulu Zhao and Hua Zhou and Donglin Hao},
year = {2024}… See the full description on the dataset page: https://huggingface.co/datasets/BAAI/IndustryCorpus2_computer_programming_code.task692_mmmlu_answer_generation_computer_security
Dataset Card for Natural Instructions (https://github.com/allenai/natural-instructions) Task: task692_mmmlu_answer_generation_computer_security
Additional Information
Citation Information
The following paper introduces the corpus in detail. If you use the corpus in published work, please cite it:
@misc{wang2022supernaturalinstructionsgeneralizationdeclarativeinstructions,
title={Super-NaturalInstructions: Generalization via Declarative Instructions on… See the full description on the dataset page: https://huggingface.co/datasets/Lots-of-LoRAs/task692_mmmlu_answer_generation_computer_security.tess-atari-15hz-384
TESS-Atari Stage 1 - Preprocessed (15Hz, 384x384)
Training-ready version of the 15Hz dataset with images pre-resized to 384x384 (SmolVLM native resolution).
Overview
Metric
Value
Source
TESS-Computer/atari-vla-stage1-15hz
Samples
1,340,293
Image Size
384x384 (pre-resized)
Action Rate
15 Hz (3 actions per observation)
Format
Lumine-style action tokens
Why Preprocessed?
Training VLMs requires resizing images to the model's native… See the full description on the dataset page: https://huggingface.co/datasets/TESS-Computer/tess-atari-15hz-384.computer-vision
Hibou Computer Vision Dataset
The Hibou Project is a drone recognition and localization system.
It is designed to detect and localize drones in real-time, using a combination of audio and video.
Official code repo: Hibou Project
This dataset is designed to train YOLO-based models for drone detection.
Object Classes
ID
Class Name
Ratio
0
Drone
74.17%
1
Other
25.83%
Dataset Description
Column
Description
image
Image from the… See the full description on the dataset page: https://huggingface.co/datasets/Hibou-Foundation/computer-vision.task701_mmmlu_answer_generation_high_school_computer_science
Dataset Card for Natural Instructions (https://github.com/allenai/natural-instructions) Task: task701_mmmlu_answer_generation_high_school_computer_science
Additional Information
Citation Information
The following paper introduces the corpus in detail. If you use the corpus in published work, please cite it:
@misc{wang2022supernaturalinstructionsgeneralizationdeclarativeinstructions,
title={Super-NaturalInstructions: Generalization via Declarative… See the full description on the dataset page: https://huggingface.co/datasets/Lots-of-LoRAs/task701_mmmlu_answer_generation_high_school_computer_science.Computer-Science-Pretrainingcomputer-use
Computer Use Trajectories
Successful computer-use agent trajectories collected on OSWorld tasks.
Dataset Details
Rows: 160 (one per task trajectory)
Steps: 1,378 total across all trajectories (avg ~8.6 steps/task)
Agent: Gemini 3 Flash Preview with linearized accessibility-tree grounding
Score filter: Only trajectories with score = 1.0 (fully successful)
Domains
Domain
Tasks
Description
chrome
21
Web browsing tasks in Google Chrome
gimp
15
Image… See the full description on the dataset page: https://huggingface.co/datasets/REXX-NEW/computer-use.csgo-vla-stage1-16hz
CS:GO VLA Stage 1 Dataset (16Hz)
Vision-Language-Action dataset for Counter-Strike: Global Offensive, converted from the TeaPearce CS:GO dataset.
Overview
Frame rate: 16Hz (native, 1 action per frame)
Total samples: ~5.5M frames
Split: train (5M) / test (500K) following Diamond split
Map: Dust2 deathmatch
Action Format
<|action_start|> mouse_x mouse_y [keys] <|action_end|>
Examples:
<|action_start|> 0 0 <|action_end|> # idle… See the full description on the dataset page: https://huggingface.co/datasets/TESS-Computer/csgo-vla-stage1-16hz.mmlu-computer_security-neg-prepend-verbal
Dataset Card for "mmlu-computer_security-neg-prepend-verbal"
More Information needed
mmlu-computer_security-neg-prepend
Dataset Card for "mmlu-computer_security-neg-prepend"
More Information needed
vidore_v3_computer_science_embeddingNOTE
ViDoRe V3: Computer Science dataset ColQwen2 Embeddings
This dataset contains pre-computed embeddings for the ViDoRe V3 : Computer Science dataset using the ColQwen2 model.
ViDoRe V3 : Computer Science
This dataset, Computer Science, is a corpus of textbooks from the openstacks website, intended for long-document understanding tasks. It is one of the 10 corpora comprising the ViDoRe v3 Benchmark.
About ViDoRe v3
ViDoRe V3 is our latest benchmark for RAG evaluation on… See the full description on the dataset page: https://huggingface.co/datasets/WenxingZhu/vidore_v3_computer_science_embedding.mmlu-computer_security-neg
Dataset Card for "mmlu-computer_security-neg"
More Information needed
mmlu-computer_security-neg-answer
Dataset Card for "mmlu-computer_security-neg-answer"
More Information needed
mmlu-computer_security-neg-prepend-fix
Dataset Card for "mmlu-computer_security-neg-prepend-fix"
More Information needed
je-veux-un-pack-de-computer-vision-pour-detecter-les-objets
Je veux un pack de computer vision pour detecter les objets,…
Je veux un pack de computer vision pour detecter les objets, dans des salles de bains pour le segment home avec des depth map
This dataset mirrors public data-pack render outputs from Physicl.
Each row represents one render view. The image column contains a stable URL to the primary render image uploaded under /data; image_path stores the relative repository path and data_commit_sha pins the Hugging Face dataset… See the full description on the dataset page: https://huggingface.co/datasets/physicl-test/je-veux-un-pack-de-computer-vision-pour-detecter-les-objets.hello-je-veux-un-pack-de-computer-vision-pour-detecter-les
Hello, Je veux un pack de computer vision pour detecter les …
Hello, Je veux un pack de computer vision pour detecter les objets, dans des salles de bains pour le segment home avec des depth map, 1 camera, 1 render
This dataset mirrors public data-pack render outputs from Physicl.
Each row represents one render view. The image column contains a stable URL to the primary render image uploaded under /data; image_path stores the relative repository path and data_commit_sha pins the… See the full description on the dataset page: https://huggingface.co/datasets/physicl-test/hello-je-veux-un-pack-de-computer-vision-pour-detecter-les.mmlu-computer_security-rule-neg-prepend
Dataset Card for "mmlu-computer_security-rule-neg-prepend"
More Information needed