nvidia/PhysicalAI-Event-Videos

1---2license: other3license_name: nvidia-dataset-license4license_link: >-5  https://assets.ngc.nvidia.com/products/api-catalog/legal/Personas-Dataset-NVIDIA-Training-Dataset-License-Agreement-2025-11-26.pdf6pretty_name: PhysicalAI-Event-Videos7task_categories:8- video-classification9- video-text-to-text10language:11- en12size_categories:13- 1K<n<10K14tags:15- video-anomaly-detection16- safety-events17- surveillance18- synthetic19- text-to-video-retrieval20---21 22# PhysicalAI-Event-Videos23 24PhysicalAI-Event-Videos is a video-anomaly and event dataset for developing text-to-video anomaly-search and safety-event-understanding systems. Version 1.0 provides structured annotations for 1,612 parent-video records and 3,796 labeled chunks, with 71,023 captions and queries across person-attribute, general-caption, anomaly, and action search tasks.25 26The redistributable media payload contains 1,486 NVIDIA-generated parent videos and 2,939 packaged chunk clips. Of those chunk clips, 2,919 have retained labels and 20 remain in the media archive without labels. CHAD and VAD-R1 media are not redistributed; the annotation index includes labels and public source pointers for those datasets.27 28## Dataset highlights29 30![A 3-by-3 grid of representative NVIDIA-generated normal and nonviolent event videos.](physicalai_event_videos_highlights_3x3.gif)31 32The montage shows nine representative NVIDIA-generated videos spanning normal activity and nonviolent retail, workplace, and public-space events. It includes selected examples from NV-Retail-Shoplifting-v0.1 and does not include CHAD or VAD-R1 source media.33 34## Dataset description35 36The dataset covers human-activity anomalies such as falling, fighting, assault, climbing, throwing objects, shoplifting, and vaping or smoking. It also includes warehouse and industrial-safety events such as forklift incidents, forklift-person near misses, falling or damaged boxes, fire and smoke, liquid spills, and personal protective equipment violations.37 38Hard-negative samples depict normal or non-anomalous activity that may visually resemble an anomalous event. These samples are intended to help models distinguish true safety incidents from benign activities and reduce false-positive detections.39 40The videos span simulated environments including warehouses, retail spaces, offices, parking lots, streets, restaurants, parks, hospitals, schools, construction sites, stadiums, transit areas, airports, and banks.41 42- **Owner:** NVIDIA Corporation43- **Creation date:** June 15, 202644- **Version:** v1.045- **Split:** train only; validation and test splits are not included46- **Modalities:** video, audio, and text47 48## License and terms of use49 50License: NVIDIA Dataset License Agreement.51 52Use of this dataset is governed by the NVIDIA Dataset License Agreement. Certain videos in this dataset were generated using Google Veo 3. The Veo-generated videos also remain subject to the applicable [Google terms](https://cloud.google.com/terms/service-terms), including restrictions on using such videos to develop a similar or competing product or service or to create or improve models similar to a Google model.53 54VAD-R1 and CHAD source media are not distributed. Users must obtain those datasets directly from the applicable source under its respective terms. The Public Safety Sample Videos seed dataset is not distributed.55 56## Intended use57 58PhysicalAI-Event-Videos is intended for application developers working on video understanding and safety-related AI systems. Potential uses include:59 60- Video anomaly detection61- Safety-event classification62- Temporal anomaly localization63- Video-language model training64- Synthetic-to-real transfer research65- Hard-negative mining and false-positive reduction66- Industrial and warehouse safety monitoring research67- Retail and public-space anomaly detection research68- Multimodal audio, video, and text representation learning69 70## Dataset characterization71 72### Data collection method73 74The release combines annotations for NVIDIA-generated and public-source videos with a redistributable NVIDIA-generated media payload. NVIDIA-generated videos were created using Google Veo 3 and Cosmos 3 Super. Annotations also cover CHAD and VAD-R1, but those third-party source videos are not distributed; users must obtain them from their original sources. The Public Safety Sample Videos seed dataset is not distributed.75 76### Labeling method77 78Labeling is hybrid: synthetic, automated, and manually corrected. Event labels and descriptive metadata are derived from generation specifications and prompts. Derived video chunks and associated manifests were produced through automated preprocessing. A subset of annotations received human review and correction; remaining query and caption fields may be model-generated.79 80## Dataset format and contents81 82- **Video:** MP4 containers with H.264 video83- **Audio:** AAC audio streams embedded in MP4 files when available84- **Text:** JSON labels and provenance metadata, plus CSV manifests and summary statistics85 86```text87prepare_chunks.py                 optional reconstruction from downloaded parent videos88README.md                         this dataset card89physicalai_event_videos_highlights_3x3.gif90anomaly-category-distribution.csv91anomaly-category-mapping.json922026/93├── train/94│   ├── videos.zip                1,486 NVIDIA-generated parent videos95│   ├── chunks.zip                2,939 NVIDIA-generated chunk clips96│   ├── labels.json               1,612 parent records with 3,796 labeled chunks97│   └── video_manifest.csv        filename-to-dataset lookup98├── stats.json                    composition and per-field query counts99└── provenance.json               public source and generation provenance100```101 102The 2,939 packaged chunk clips comprise 2,919 labeled clips plus 20 retained media clips whose labels are not included. CHAD and VAD-R1 rows in `labels.json` identify source media through public pointers; their media is not present in the archives.103 104## Label schema105 106`labels.json` is a list of per-video records:107 108```jsonc109{110  "split": "train",111  "dataset": "NV-VAD-v1.1",112  "video_id": "video_01126",113  "anomaly_type": "person running",114  "total_frames": 189,115  "video_duration_sec": 7.875,116  "fps": 24.0,117  "video_path": "videos/video_01126.mp4",118  "chunks": [119    {120      "chunk_index": 0,121      "start_time_sec": 0.0,122      "end_time_sec": 5.0,123      "start_frame": 0,124      "end_frame": 120,125      "is_anomaly": true,126      "anomaly_type": "person running",127      "dense_caption": "...",128      "scene_caption": "...",129      "pas_queries": ["..."],130      "caption_queries": ["..."],131      "anomaly_queries": ["..."],132      "action_queries": ["..."],133      "chunk_file": "chunk_000.mp4"134    }135  ]136}137```138 139### Joining labels to media140 141| Requested media | Path inside the extracted archive |142|---|---|143| Parent video | `videos/` + `basename(video_path)` |144| Chunk clip | `chunks/<dataset>/<video_id>/<chunk_file>` |145 146Parent-video basenames are unique across the redistributable NVIDIA-generated videos. Chunk filenames are not unique on their own, so chunk paths are nested by `dataset` and `video_id`.147 148The 1,486 NVIDIA-generated parent videos use a deterministic global sequence, `video_00001.mp4` through `video_01486.mp4`. The filename does not encode its source; use `dataset` in `labels.json` or `video_manifest.csv` to distinguish releases.149 150## Annotation examples151 152Each labeled chunk contains temporal bounds, an anomaly decision and free-text anomaly type, two complementary captions, and four query sets. The example below is an excerpt from an NVIDIA-generated forklift-person near-miss chunk.153 154| Annotation type | Example |155|---|---|156| Temporal localization | `0.0-5.0 seconds` |157| Anomaly flag | `is_anomaly: true` |158| Anomaly type | `near miss forklift person` |159| Dense caption | A forklift drives down a warehouse aisle and nearly collides with two employees standing in its path, forcing them to step aside quickly. 00:00-00:02: Two warehouse employees in safety vests stand in the aisle discussing a clipboard while a forklift approaches from the distance. 00:02-00:04: The forklift continues to drive forward toward the two employees, who remain stationary and focused on their documents. 00:04-00:07: As the forklift closes the distance rapidly, the employees notice it and quickly step out of the way to avoid a collision. |160| Scene caption | The scene takes place indoors within a large warehouse aisle, flanked by towering orange industrial shelving units filled with boxed inventory. The camera is positioned at a fixed, eye-level angle looking down the center of the aisle. The area is brightly lit with cool, overhead industrial lighting, and the general activity involves personnel in safety vests coordinating logistics and operating a forklift. |161| Person-attribute search (PAS) | `person wearing yellow vest and grey shirt`; `person wearing yellow vest and black pants` |162| General caption query | `warehouse aisle with orange industrial shelving units`; `two employees discussing a clipboard in a warehouse` |163| Anomaly query | `person in yellow safety vest nearly hit by a forklift`; `forklift almost colliding with warehouse employees` |164| Action query | `a forklift driving`; `people standing in a path` |165 166`anomaly_type` remains free text. The derived anomaly families below are provided for summary reporting and do not replace the original labels.167 168## Dataset quantification169 170### Annotation coverage171 172- **Parent records:** 1,612173- **Labeled chunks:** 3,796174- **Anomalous labeled chunks:** 2,483175- **Normal labeled chunks:** 1,313176- **Captions and queries:** 71,023 total177 178### Query counts179 180| Query field | Count |181|---|---:|182| Person-attribute search (`pas_queries`) | 18,103 |183| General caption (`caption_queries`) | 18,823 |184| Anomaly (`anomaly_queries`) | 18,913 |185| Action (`action_queries`) | 15,184 |186| **Total** | **71,023** |187 188### Source coverage189 190| Source dataset | Parent records | Labeled chunks | Media availability in this release |191|---|---:|---:|---|192| `CHAD` | 46 | 392 | Source pointers only |193| `NV-Retail-Shoplifting-v0.1` | 200 | 399 | NVIDIA-generated media included |194| `NV-VAD-v1` | 925 | 1,799 | NVIDIA-generated media included |195| `NV-VAD-v1.1` | 361 | 721 | NVIDIA-generated media included |196| `VAD-R1` | 80 | 485 | Source pointers only |197| **Total** | **1,612** | **3,796** | |198 199### Redistributable NVIDIA-generated media200 201- **Parent videos:** 1,486202- **Labeled chunk clips:** 2,919203- **Retained but unlabeled chunk clips:** 20204- **Packaged chunk clips:** 2,939205 206## Anomaly-category distribution207 208`anomaly_type` is a free-text field rather than a closed categorical label. For reporting, each of the 2,483 anomalous chunks is assigned to one mutually exclusive family using a documented, deterministic first-match mapping. These derived families do not replace the original labels.209 210| Derived anomaly family | Labeled chunks | Share of anomalous chunks |211|---|---:|---:|212| Fall / collapse | 379 | 15.3% |213| Fight / assault | 340 | 13.7% |214| Smoking / vaping | 282 | 11.4% |215| Climbing / jumping | 201 | 8.1% |216| Throwing / vandalism | 164 | 6.6% |217| Running | 160 | 6.4% |218| Fire / smoke / explosion | 145 | 5.8% |219| Falling / damaged objects | 137 | 5.5% |220| Traffic / parking safety | 125 | 5.0% |221| Retail theft / concealment | 111 | 4.5% |222| Forklift incident | 91 | 3.7% |223| Liquid spill / leak | 91 | 3.7% |224| Animal-related hazard | 91 | 3.7% |225| Generic warehouse-safety violation | 82 | 3.3% |226| PPE violation | 32 | 1.3% |227| Weather / environmental hazard | 27 | 1.1% |228| Other / ambiguous | 25 | 1.0% |229 230Percentages use anomalous labeled chunks as the denominator and may not sum to exactly 100% because values are rounded to one decimal place. See [`anomaly-category-distribution.csv`](anomaly-category-distribution.csv) for the summary table and [`anomaly-category-mapping.json`](anomaly-category-mapping.json) for the ordered mapping rules.231 232## Preparing chunks from downloaded videos233 234`chunks.zip` is ready to use. If parent videos are downloaded separately, the optional `prepare_chunks.py` script reconstructs labeled time windows without making network requests:235 236```bash237python3 prepare_chunks.py \238  --labels 2026/train/labels.json \239  --videos-dir /path/to/downloaded/videos \240  --output-dir 2026/train/chunks \241  --workers 4242```243 244The script searches the video directory recursively, reads exact start and end times from `labels.json`, and writes `chunks/<dataset>/<video_id>/<chunk_file>`. It requires Python 3, `ffmpeg`, and `ffprobe`. Use `--dry-run` to validate inputs or `--dataset NAME` to prepare a subset. Reconstructed clips match the labeled time windows but may not be byte-identical across ffmpeg versions.245 246## Public source pages247 248These are public third-party landing pages, not NVIDIA-controlled storage:249 250- **CHAD:** [official repository](https://github.com/TeCSAR-UNCC/CHAD) and [primary video archive](https://drive.google.com/file/d/13am4hfhicErcozAYgkmQm02_K-cCmtkQ/view?usp=sharing)251- **VAD-R1:** [public Hugging Face dataset](https://huggingface.co/datasets/wbfwonderful/Vad-R1)252 253CHAD and VAD-R1 media must be obtained from their original sources under the applicable terms. This release does not redistribute that media.254 255## Known limitations256 2571. **Third-party source dependency.** CHAD and VAD-R1 annotations require users to obtain the corresponding media separately and comply with each source dataset's terms.2582. **Model-generated text.** A subset of annotations received human review and correction, while remaining query and caption fields may contain model errors or inconsistencies.2593. **Free-text anomaly types.** `anomaly_type` is not a closed ontology. The supplied category mapping is a reporting aid based on deterministic text matching.2604. **Near-duplicate scenarios.** Many NVIDIA-generated videos share related generation scenarios. Split by generation scenario, not only by `video_id`, to reduce train-evaluation leakage.2615. **Synthetic-to-real gap.** Models trained on simulated media may not transfer directly to real environments, camera systems, or populations without additional validation.262 263## Ethical considerations264 265NVIDIA believes Trustworthy AI is a shared responsibility and has established policies and practices to enable development for a wide array of AI applications. Developers should work with their organizations to ensure this dataset meets the requirements of the relevant industry and use case and addresses unforeseen product misuse.266 267Users should evaluate performance, safety, fairness, privacy, and failure modes in the intended deployment context. Synthetic data may still encode artifacts or biases from prompts and generation systems, and third-party annotations should be assessed with their source context and terms.268