mOONIm/PhysicalAI-SmartSpaces

Physical AI Smart Spaces Dataset

[image]

Overview

Comprehensive, annotated dataset for multi-camera tracking and 2D/3D object detection. This dataset is synthetically generated with Omniverse.

This dataset consists of over 250 hours of video from across nearly 1,500 cameras from indoor scenes in warehouses, hospitals, retail, and more. The dataset is time synchronized for tracking humans across multiple cameras using feature representation and no personal data.

Dataset Description

Dataset Owner(s)

NVIDIA

Dataset Creation Date

We started to create this dataset in December, 2023. First version was completed and released as part of 8th AI City Challenge in conjunction with CVPR 2024.

Dataset Characterization

• —Data Collection Method: Synthetic
• —Labeling Method: Automatic with IsaacSim

Video Format

• —Video Standard: MP4 (H.264)
• —Video Resolution: 1080p
• —Video Frame rate: 30 FPS

Ground Truth Format (MOTChallenge) for MTMC_Tracking_2024

Annotations are provided in the following text format per line:

<camera_id> <obj_id> <frame_id> <xmin> <ymin> <width> <height> <xworld> <yworld>

• —<camera_id>: Numeric identifier for the camera.
• —<obj_id>: Consistent numeric identifier for each object across cameras.
• —<frame_id>: Frame index starting from 0.
• —<xmin> <ymin> <width> <height>: Axis-aligned bounding box coordinates in pixels (top-left origin).
• —<xworld> <yworld>: Global coordinates (projected bottom points of objects) based on provided camera matrices.

The video file and calibration (camera matrix and homography) are provided for each camera view.

Calibration and ground truth files in the updated 2025 JSON format are now also included for each scene.

Notes:

• —Some calibration fields—such as camera coordinates, camera directions, and scale factors—are not be available for the 2024 dataset due to original data limitations.
• —Please be aware that the video identified as scene_071/camera_0649 has encountered corruption issues. We advise you to exclude this video from your submissions.
• —In the sequences ranging from scenes 071 to 080, you'll find a storage room distinct from the primary retail space. Although separate, individuals can access this area. We have ensured that these sequences are synchronized effectively, treating them as part of the same continuous space for analysis purposes.

Directory Structure for MTMC_Tracking_2025

• —videos/: Video files.
• —depth_maps/: Depth maps stored as PNG images and compressed within HDF5 files. These files are exceedingly large; you may choose to use RGB videos only if preferred.
• —ground_truth.json: Detailed ground truth annotations (see below).
• —calibration.json: Camera calibration and metadata.
• —map.png: Visualization map in top-down view.

Ground Truth Format (JSON) for MTMC_Tracking_2025

Annotations per frame:

{
  "<frame_id>": [
    {
      "object_type": "<class_name>",
      "object_id": <int>,
      "3d_location": [x, y, z],
      "3d_bounding_box_scale": [w, l, h],
      "3d_bounding_box_rotation": [pitch, roll, yaw],
      "2d_bounding_box_visible": {
        "<camera_id>": [xmin, ymin, xmax, ymax]
      }
    }
  ]
}

Calibration Format (JSON) for MTMC_Tracking_2025

Contains detailed calibration metadata per sensor:

{
  "calibrationType": "cartesian",
  "sensors": [
    {
      "type": "camera",
      "id": "<sensor_id>",
      "coordinates": {"x": float, "y": float},
      "scaleFactor": float,
      "translationToGlobalCoordinates": {"x": float, "y": float},
      "attributes": [
        {"name": "fps", "value": float},
        {"name": "direction", "value": float},
        {"name": "direction3d", "value": "float,float,float"},
        {"name": "frameWidth", "value": int},
        {"name": "frameHeight", "value": int}
      ],
      "intrinsicMatrix": [[f_x, 0, c_x], [0, f_y, c_y], [0, 0, 1]],
      "extrinsicMatrix": [[3×4 matrix]],
      "cameraMatrix": [[3×4 matrix]],
      "homography": [[3×3 matrix]]
    }
  ]
}

Evaluation

• —2024 Edition: Evaluation based on 3D-location-based HOTA scores at the 2024 AI City Challenge Server. The submission is currently disabled, as the ground truths of test set are provided with this release.
• —2025 Edition: Evaluation based on 3D-bounding-box-based HOTA scores at the 2025 AI City Challenge Server.

Scene ID Table for MTMC_Tracking_2025

Dataset Quantification

References

Please cite the following papers when using this dataset:

@InProceedings{Tang25AICity25,
author = {Zheng Tang and Shuo Wang and David C. Anastasiu and Ming-Ching Chang and Anuj Sharma and Quan Kong and Norimasa Kobori and Munkhjargal Gochoo and Ganzorig Batnasan and Munkh-Erdene Otgonbold and Fady Alnajjar and Jun-Wei Hsieh and Tomasz Kornuta and Xiaolong Li and Yilin Zhao and Han Zhang and Subhashree Radhakrishnan and Arihant Jain and Ratnesh Kumar and Vidya N. Murali and Yuxing Wang and Sameer Satish Pusegaonkar and Yizhou Wang and Sujit Biswas and Xunlei Wu and Zhedong Zheng and Pranamesh Chakraborty and Rama Chellappa},
title = {The 9th {AI} {C}ity {C}hallenge},
booktitle = {Proc. ICCV Workshops},
pages = {5467--5476},
address = {Honolulu, HI, USA},
year = {2025}
}

@inproceedings{Wang24AICity24,
author = {Shuo Wang and David C. Anastasiu and Zheng Tang and Ming-Ching Chang and Yue Yao and Liang Zheng and Mohammed Shaiqur Rahman and Meenakshi S. Arya and Anuj Sharma and Pranamesh Chakraborty and Sanjita Prajapati and Quan Kong and Norimasa Kobori and Munkhjargal Gochoo and Munkh-Erdene Otgonbold and Ganzorig Batnasan and Fady Alnajjar and Ping-Yang Chen and Jun-Wei Hsieh and Xunlei Wu and Sameer Satish Pusegaonkar and Yizhou Wang and Sujit Biswas and Rama Chellappa},
title = {The 8th {AI} {C}ity {C}hallenge},
booktitle = {Proc. CVPR Workshops},
pages = {7261--7272},
address = {Seattle, WA, USA},
year = {2024}
}

@misc{Wang24MCBLT,
author = {Yizhou Wang and Tim Meinhardt and Orcun Cetintas and Cheng-Yen Yang and Sameer Satish Pusegaonkar and Benjamin Missaoui and Sujit Biswas and Zheng Tang and Laura Leal-Taix{\'e}},
title = {{MCBLT}: {M}ulti-Camera Multi-Object {3D} Tracking in Long Videos},
note = {arXiv:2412.00692},
year = {2024}
}

Ethical Considerations

NVIDIA believes Trustworthy AI is a shared responsibility and we have established policies and practices to enable development for a wide array of AI applications. When downloaded or used in accordance with our terms of service, developers should work with their internal model team to ensure this model meets requirements for the relevant industry and use case and addresses unforeseen product misuse.

Please report security vulnerabilities or NVIDIA AI Concerns here.

Changelog

• —2025-07-14: Provided evaluation tools for both 2024 and 2025 datasets.
• —2025-06-16: Corrected rotation and misalignment issues in ground truth and calibration for MTMC_Tracking_2025 scenes.
• —2025-04-27: Added depth maps to all MTMC_Tracking_2025 scenes.
• —2025-04-23: Added 2025-format calibration and ground truth JSON files to all MTMC_Tracking_2024 scenes.
• —2025-05-19: Updated corrupted files (videos and depth maps) in train/val scenes of MTMC_Tracking_2025.
• —2025-05-28: Added 4 test scenes of MTMC_Tracking_2025 and provided scene ID table.
• —2025-06-16: Updated ground truth and calibration files, and removed misaligned videos and depth maps for train/val scenes in MTMC_Tracking_2025.