nvidia/difix_ref
79.1k
1---2datasets:3- DL3DV/DL3DV-10K-Sample4language:5- en6---7# **Difix3D+: Improving 3D Reconstructions with Single-Step Diffusion Models** 8CVPR 2025 (Oral) 9[**Code**](https://github.com/nv-tlabs/Difix3D) | [**Project Page**](https://research.nvidia.com/labs/toronto-ai/difix3d/) | [**Paper**](https://arxiv.org/abs/2503.01774)10 11## Description: 12Difix is a single-step image diffusion model trained to enhance and remove artifacts in rendered novel views caused by13underconstrained regions of 3D representation. The technology behind Difix is based on the concepts outlined in the paper titled14[DIFIX3D+: Improving 3D Reconstructions with Single-Step Diffusion Models](https://arxiv.org/abs/2503.01774 ).15 16Difix has two operation modes: 17 18* Offline mode: Used during the reconstruction phase to clean up pseudo-training views that are rendered from the reconstruction19 and then distill them back into 3D. This greatly enhances underconstrained regions and improves the overall 3D representation quality. 20* Online mode: Acts as a neural enhancer during inference, effectively removing residual artifacts arising from imperfect 3D21 supervision and the limited capacity of current reconstruction models. 22 23Difix is an all-encompassing solution, a single model compatible for both NeRF and 3DGS representations.24 25**This model is ready for research and development/non-commercial use only.**26 27**Model Developer:** NVIDIA28 29**Model Versions:** difix_ref30 31**Deployment Geography:** Global32 33### License/Terms of Use:34The use of the model and code is governed by the NVIDIA License. Additional Information: [LICENSE.md · stabilityai/sd-turbo at main](https://huggingface.co/stabilityai/sd-turbo/blob/main/LICENSE.md)35 36 37### Use Case:38Difix is intended for Physical AI developers looking to enhance and improve their Neural Reconstruction pipelines. The model takes an image as an input and outputs a fixed image39 40**Release Date:** Github: [June 2025](https://github.com/nv-tlabs/Difix3D)41 42## Model Architecture43 44**Architecture Type**: UNet45 46**Network Architecture**: A latent diffusion-based UNet coupled with a variational autoencoder (VAE).47 48## Input49 50**Input Type(s)**: Image51 52**Input Format(s)**: Red, Green, Blue (RGB)53 54**Input Parameters**: Two-Dimensional (2D)55 56**Other Properties Related to Input**:57* Specific Resolution: [576px x 1024px]58 59## Output60 61**Output Type(s)**: Image62 63**Output Format(s)**: Red, Green, Blue (RGB)64 65**Output Parameters**: Two-Dimensional (2D)66 67**Other Properties Related to Output**:68* Specific Resolution: [576px x 1024px]69 70## Software Integration71 72**Runtime Engine(s)**: PyTorch73 74**Supported Hardware Microarchitecture Compatibility**:75* NVIDIA Ampere76* NVIDIA Hopper77 78**Note**: We are testing with FP32 Precision.79 80## Inference81**Acceleration Engine**: [PyTorch](https://pytorch.org/)82 83**Test Hardware**: 84* A10085* H10086 87**Operating System(s):** Linux (We have not tested on other operating systems.)88 89**System Requirements and Performance:**90This model requires X GB of GPU VRAM.91The following table shows inference time for a single generation across different NVIDIA GPU hardware:92 93| GPU Hardware | Inference Runtime |94|--------------|----------------------------|95| NVIDIA A100 | 0.355 sec |96| NVIDIA H100 | 0.223 sec |97 98## Use the Difix Model99Please visit the [Difix3D repository](https://github.com/nv-tlabs/Difix3D) to access all relevant files and code needed to use Difix100 101 102## Difix Dataset103- Data Collection Method: Human104- Labeling Method by Dataset: Human105- Properties: Difix was trained, tested, and evaluated using the [DL3DV-10k dataset](https://huggingface.co/datasets/DL3DV/DL3DV-10K-Sample), where 80% of the data was used for training, 10% for evaluation, and 10% for testing. DL3DV-10K is a large-scale dataset consisting of 10,510 high-resolution (4K) real-world video sequences, totaling approximately 51.2 million frames. The scenes span 65 diverse categories across indoor and outdoor environments. Each video is accompanied by metadata describing environmental conditions such as lighting (natural, artificial, mixed), surface materials (e.g., reflective or transparent), and texture complexity. The dataset is designed to support the development and evaluation of learning-based 3D vision methods.106 107 108## Ethical Considerations:109NVIDIA believes Trustworthy AI is a shared responsibility and we have established policies and practices to enable development for a wide array of AI applications. When downloaded or used in accordance with our terms of service, developers should work with their internal model team to ensure this model meets requirements for the relevant industry and use case and addresses unforeseen product misuse. 110 111Please report security vulnerabilities or NVIDIA AI Concerns [here](https://www.nvidia.com/en-us/support/submit-security-vulnerability/)112 113---114 115## ModelCard++116 117### Bias118 119| Field | Response |120| :--------------------------------------------------------------------------------------------------------------------------------------------------------------- | :------- |121| Participation considerations from adversely impacted groups [protected classes](https://www.senate.ca.gov/content/protected-classes) in model design and testing: | None |122| Measures taken to mitigate against unwanted bias: | None |123 124### Explainability125 126| Field | Response |127| :-------------------------------------------------------- | :------------------------------------------------------------------------------------------------------------------- |128| Intended Domain: | Advanced Driver Assistance Systems |129| Model Type: | Image-to-Image |130| Intended Users: | Autonomous Vehicles developers enhancing and improving Neural Reconstruction pipelines. |131| Output: | Image |132| Describe how the model works: | The model takes as an input an image, and outputs a fixed image |133| Name the adversely impacted groups this has been tested to deliver comparable outcomes regardless of: | None |134| Technical Limitations: | The reconstruction relies on the quality and consistency of input images and camera calibrations; any deficiencies in these areas can negatively impact the final output. |135| Verified to have met prescribed NVIDIA quality standards: | Yes |136| Performance Metrics: | FID (Fréchet Inception Distance), PSNR (Peak Signal-to-Noise Ratio), LPIPS (Learned Perceptual Image Patch Similarity) |137| Potential Known Risks: | The model is not guaranteed to fix 100% of the image artifacts. please verify the generated scenarios are context and use appropriate. |138| Licensing: | The use of the model and code is governed by the NVIDIA License. Additional Information: [LICENSE.md · stabilityai/sd-turbo at main](https://huggingface.co/stabilityai/sd-turbo/blob/main/LICENSE.md). |139 140### Privacy141 142| Field | Response |143| :------------------------------------------------------------------ | :------------- |144| Generatable or reverse engineerable personal data? | No |145| Personal data used to create this model? | No |146| How often is the dataset reviewed? | Before release |147| Is there provenance for all datasets used in training? | Yes |148| Does data labeling (annotation, metadata) comply with privacy laws? | Yes |149| Is data compliant with data subject requests for data correction or removal, if such a request was made? | Yes |150 151### Safety & Security152 153| Field | Response |154| :---------------------------------------------- | :----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |155| Model Application(s): | Image Enhancement |156| List types of specific high-risk AI systems, if any, in which the model can be integrated: | The model can be used to develop Autonomous Vehicles stacks that can be integrated inside vehicles. The Difix model should not be deployed in a vehicle. |157| Describe the life critical impact (if present). | N/A - The model should not be deployed in a vehicle and will not perform life-critical tasks. |158| Use Case Restrictions: | Your use of the model and code is governed by the NVIDIA License. Additional Information: LICENSE.md · stabilityai/sd-turbo at main |159| Model and dataset restrictions: | The Principle of least privilege (PoLP) is applied limiting access for dataset generation and model development. Restrictions enforce dataset access during training, and dataset license constraints adhered to. |160 161## Support162 163**DiFix is a previous-generation model:** please use [Fixer](https://github.com/nv-tlabs/Fixer) for active development and support.164 165**Usage questions and discussion:** please post on the [NVIDIA Developer Forum (Omniverse / NuRec)](https://forums.developer.nvidia.com/c/omniverse/platform/nurec/752).166 167**Code-level bugs, documentation issues, and feature requests:** file a [GitHub issue](https://github.com/nv-tlabs/Fixer/issues/new/choose) for Fixer using the appropriate template.168 169**Security vulnerabilities:** use [NVIDIA's Vulnerability Disclosure Program](https://app.intigriti.com/programs/nvidia/nvidiavdp/detail). Do not file security issues publicly in this repository.170 171*The Hugging Face Community tab for this model card will be disabled on 08/10/2026. Please use the channels above.*172