Datasetpublic

Arm/neural-graphics-dataset

Neural Graphics Dataset A compact collection of reference image sequences with accompanying motion, depth, and related rendering data, designed for training, validation, and evaluation of neural graphics models. It is intended for use with models in the Neural Graphics Model Gym, including: Neural Super Sampling (NSS) Neural Frame Rate Upscaling (NFRU) The dataset is intended as a small, practical example dataset for tutorials and experimentation. For best performance in… See the full description on the dataset page: https://huggingface.co/datasets/Arm/neural-graphics-dataset.

sourceHugging Faceotherupdated 2mo agoView on Hugging Face

4likes4.2kdownloads

Dataset Card

Neural Graphics Dataset

A compact collection of reference image sequences with accompanying motion, depth, and related rendering data, designed for training, validation, and evaluation of neural graphics models.

It is intended for use with models in the Neural Graphics Model Gym, including:

The dataset is intended as a small, practical example dataset for tutorials and experimentation. For best performance in production-level training, we recommend collecting a substantially larger dataset using a wide variety of scenes that reflect the types of content the models will be expected to handle.

The model-specific sections below provide the minimum dataset sizes used to train each model.

<table> <tr><td><strong>Curated By</strong></td><td>Arm Limited</td></tr> <tr><td><strong>License</strong></td><td><a href="https://huggingface.co/datasets/Arm/neural-graphics-dataset/blob/main/ArmAIModelCommunityLicensev10_PRE-1154.pdf">Arm AI Model Community License</a></td></tr> </table>

Neural Super Sampling (NSS)

Dataset Description

The current dataset includes a compact collection of data for NSS. Its primary purpose is to demonstrate the NSS model development workflow in the Neural Graphics Model Gym. To train NSS, we use a dataset containing more than 50,000 frames. In general, larger datasets improve machine learning model performance.

A plugin for capturing data in Unreal® Engine is available to capture your own content for use in (re)training or evaluation in the Neural Graphics Model Gym.

Dataset Sources

The NSS dataset was captured from the Amazon Lumberyard Bistro scene implemented in Unreal Engine.

—Original asset location: Morgan McGuire, Computer Graphics Archive, July 2017

Dataset Structure

The NSS dataset uses sequence-level Safetensors files for training, validation, and testing. Each file stores one full captured sequence rather than one single frame.

neural-graphics-dataset/
└── nss/
  ├── test/   # test set
  ├── train/  # training set
  └── val/    # validation set

Safetensors Contents

Each NSS Safetensors file contains the following data:

Tensor	Description
`colour_linear`	Low-resolution color data in linear format
`ground_truth_linear`	High-resolution color data in linear format
`depth`	Per-pixel depth data
`depth_params`	Additional depth parameters
`motion_lr`	Low-resolution motion vectors
`motion`	High-resolution motion vectors
`exposure`	Exposure values
`jitter`	Jitter vector
`render_size`	Low-resolution image shape
`zNear`	Camera near plane values
`zFar`	Camera far plane values

The training and validation sets use cropped, reduced-resolution data, while the test set is provided at the original capture resolution.

Bias, Risks, and Limitations

The training set contains a limited number of frames to be used as training data. It is intended for use with NSS tutorials and experimentation with the NSS algorithm.

Related Resources

A pre-trained Neural Super Sampling model can be accessed from the NSS model card.

Neural Frame Rate Upscaling (NFRU)

Dataset Description

The current dataset includes a compact collection of data for NFRU. Its primary purpose is to demonstrate the NFRU model development workflow in the Neural Graphics Model Gym. To train NFRU, we use a dataset containing more than 16,000 frames. In general, larger datasets improve machine learning model performance.

Future releases of the Neural Graphics Model Gym will provide tools to capture and convert content for use in model training and retraining.

Dataset Sources

The NFRU dataset was captured from the Amazon Lumberyard Bistro scene implemented in Unreal Engine.

—Original asset location: Morgan McGuire, Computer Graphics Archive, July 2017

Dataset Structure

The NFRU dataset uses sequence-level Safetensors files for training, validation, and testing. Each file stores one full captured sequence rather than one single frame.

text

neural-graphics-dataset/
└── nfru/
  ├── test/   # test set
  ├── train/  # training set
  └── val/    # validation set

Safetensors Contents

Each NFRU Safetensors file contains the following data:

Tensor	Description
`rgb_linear`	High-resolution linear RGB frames
`rgb_reinhard`	High-resolution Reinhard-tonemapped RGB frames
`depth`	Low-resolution depth buffer
`DepthParams`	Depth reprojection parameters
`NearPlane`	Camera near plane values
`FarPlane`	Camera far plane values
`infinite_zFar`	Flag indicating the original capture used an infinite far plane
`FovX`	Horizontal camera field of view in radians
`FovY`	Vertical camera field of view in radians
`ViewProj`	4x4 view-projection matrix
`InverseY`	Flag indicating whether Y inversion should be applied to camera transforms
`exposure`	Exposure values
`jitter`	Jitter offsets
`render_size`	Render resolution
`img_id`	Frame index within the sequence
`seq_id`	Sequence identifier
`mv_{}_f30_m1`	Motion vectors to the previous 30 FPS capture
`mv_{}_f60_m1`	Motion vectors to the previous 60 FPS frame
`sy_{}_f30_m1`	Synthetic motion vectors to the previous 30 FPS capture
`sy_{}_f30_p1`	Synthetic motion vectors to the next 30 FPS capture
`sy_{}_f60_m1`	Synthetic motion vectors to the previous 60 FPS frame
`sy_{}_f60_p1`	Synthetic motion vectors to the next 60 FPS frame
`dilated_sy_{}_f30_m1`	Dilated synthetic motion vectors to the previous 30 FPS capture
`dilated_sy_{}_f30_p1`	Dilated synthetic motion vectors to the next 30 FPS capture

Tensor Naming Convention

NFRU frame indices are measured on a 60 FPS timeline and use an interpolation triplet:

—m1: previous input frame
—t: frame to be predicted
—p1: next input frame

For example, if t is frame 0004, then:

—m1 is frame 0003
—p1 is frame 0005

Motion-like tensors (mv_*, sy_*) also include an f60 or f30 tag:

—f60: motion between adjacent 60 FPS frames
—f30: motion between adjacent 30 FPS captures

60 FPS frame index	`0000`	`0001`	`0002`	`0003`	`0004`	`0005`	`0006`
60 FPS capture	x	x	x	x	x	x	x
30 FPS capture	x		x		x		x

In the stored Safetensors files, {} represents the source frame index. For example, if the source frame is 0004, then sy_{}_f30_m1 refers to motion associated with frame 0002, while sy_{}_f30_p1 refers to motion associated with frame 0006.

Bias, Risks, and Limitations

The training set contains a limited number of frames to be used as training data. It is intended for use with NFRU tutorials and experimentation with the NFRU algorithm.

Related Resources

A pre-trained Neural Frame Rate Upscaling model can be accessed from the NFRU model card.

License

The Neural Graphics Dataset is released under Arm AI Model Community License

Trademark notice

Arm® is a registered trademark of Arm Limited (or its subsidiaries) in the US and/or elsewhere.

Unreal® is a trademark or registered trademark of Epic Games, Inc. in the United States of America and elsewhere.