Harmonizer-Dataset

# HARMONIZER DATASET ## Dataset Description Training dataset for [DiffusionHarmonizer](https://research.nvidia.com/labs/sil/projects/diffusion-harmonizer/): a generative AI model for image and video enhancement bridging neural reconstruction and photorealistic simulation . Model checkpoints: [https://huggingface.co/nvidia/Harmonizer/](https://huggingface.co/nvidia/Harmonizer) Training code: [https://github.com/NVIDIA/harmonizer/](https://github.com/NVIDIA/harmonizer/) The dataset was curated to support the following functions of the model: 1. 3D reconstruction artifact removal 2. Harmonization of inserted objects to blend into the surrounding environment: a. realistic change of color and brightness b. realistic relighting c. realistic shadow insertion Dataset folder structure follows this split: 1. artifact_correction 2. ISP_modification 3. Relighting 4. shadow_asset_reinsertion 5. shadow_PBR Dataset consists of training pairs of images to support supervised learning: 1. an original recorded driving image / target look image 2. the modified counterpart to simulate input image characteristics (modification of lighting and color, introduction of 3D reconstruction artifacts, shadow removal). Note: The original dataset used for training of the checkpoints released under [Harmonizer](https://huggingface.co/nvidia/Harmonizer) consisted of ~1M image pairs and could not be released fully due to legal restrictions. To mitigate that, new data pairs have been curated to replace the unreleasable components and the full dataset size has been increased to ~1.5M image pairs. Data has been anonymized. This dataset is ready for commercial/non-commercial use within the scope described by the license agreement. ## Dataset Owner(s) NVIDIA Corporation ## Dataset Creation Date Modified Last Date: 22 April 2026 ## License/Terms of Use [NVIDIA Autonomous Vehicle Dataset License Agreement](https://huggingface.co/datasets/nvidia/Harmonizer-Dataset/blob/main/LICENSE.pdf) ## Intended Usage The dataset can be used in full to train DiffusionHarmonizer-like models. Each subset can also be downloaded separately and used for training of dedicated models, such as [Difix3D](https://research.nvidia.com/labs/toronto-ai/difix3d/) style models, or shadow removal / detection models. ## Dataset Creation ### Data Collection and Processing This dataset utilizes a **Hybrid** data collection pipeline combining real-world physical captures with high-fidelity simulations. * **Automated (Real-World Sensor Capture):** Real-world driving recordings captured via physical camera sensors. * **Synthetic (Simulator Capture):** Computer-Generated Imagery (CGI) generated via the `DriveSim` environment to simulate rare edge cases and adverse weather conditions. ### Source Data Producers * Real-world data was gathered passively from fleet vehicle cameras. * Synthetic data was algorithmically generated using the DriveSim synthetic data generation engine. ### Annotation Process This dataset contains **no labels or semantic annotations**. ## Dataset Format Image frames: png / jpg For easier downloading, the image frames have been packed into tarballs ranging in size from 5GB to 70GB. After downloading and unpacking the tarballs for each subset into each named subdirectory, the directory structure should look as follows: ``` Harmonizer-dataset/ ├─ artifact_correction/ │ ├─ train_A/ │ └─ train_B/ ├─ ISP_modification/ │ ├─ train_A/ │ └─ train_B/ ├─ Rrelighting/ │ ├─ train_A/ │ └─ train_B/ ├─ shadow_asset_reinsertion/ │ ├─ train_A/ │ └─ train_B/ └─ shadow_PBR/ ├─ train_A/ └─ train_B/ ``` ## Dataset Quantification The Harmonizer dataset is divided into five subsets with pairs of identically-named images in train_A and train_B subdirectories (as shown above). Refer to the following table for image frame pair counts and size for each subset: | subset | frame pairs | size | |:-------------------------|-------------|-------:| | artifact_correction | 621,000 | 817GB | | ISP_modification | 88,495 | 136GB | | Relighting | 583,694 | 597GB | | shadow_asset_reinsertion | 24,071 | 35GB | | shadow_PBR | 77,000 | 62GB | | Total | 1,394,260 | 1647GB | The total size of the dataset is approximately 1.7TB with almost 1.4M frame pairs. ## Reference(s) Research paper: [https://research.nvidia.com/labs/sil/projects/diffusion-harmonizer/](https://research.nvidia.com/labs/sil/projects/diffusion-harmonizer/) Dataset on Hugging Face: [https://huggingface.co/datasets/nvidia/Harmonizer-Dataset](https://huggingface.co/datasets/nvidia/Harmonizer-Dataset) ## Ethical Considerations NVIDIA believes Trustworthy AI is a shared responsibility and we have established policies and practices to enable development for a wide array of AI applications. When downloaded or used in accordance with our terms of service, developers should work with their internal developer teams to ensure this dataset meets requirements for the relevant industry and use case and addresses unforeseen product misuse. Please report quality, risk, security vulnerabilities or NVIDIA AI Concerns [here](https://www.nvidia.com/en-us/support/submit-security-vulnerability/).