PhysicalAI-Autonomous-Vehicles-NuRec

# PHYSICAL AI AUTONOMOUS VEHICLES NUREC

<video controls autoplay src="https://cdn-uploads.huggingface.co/production/uploads/671aac6b5f7cde5ac3e82f26/Kv_kpnk9hqATRNxPQJeL4.qt"></video>

The PhysicalA-AutonomousVehicle-NuRec dataset contains 918 scenes of dynamic neural reconstructions. The scenes are about 20 seconds long and stored in the form of usdz files, along with respective surface meshes. The reconstructions were generated using 6 camera views (front-wide 120 deg, front-tele 30 deg, cross right/left 120 deg and rear right/left 70 deg). Users can use these 3D reconstructed driving scenes for training and testing their autonomous vehicle (AV) systems. This dataset is ready for commercial/non-commercial AV use per the license agreement.

- Data Collection Method<br>
- [Automatic/Sensors]
- [Machine-derived] <br>

- Labeling Method<br>
- [Automatic/Sensors]
- [Machine-derived] <br>

These reconstructions are derived from clips from Physical AI AV dataset at the link below via an NCore based processing workflow

- [https://huggingface.co/datasets/nvidia/PhysicalAI-Autonomous-Vehicles](https://huggingface.co/datasets/nvidia/PhysicalAI-Autonomous-Vehicles)

## Applications

Developing a model end to end for safe use in the real world requires a combination of massive domain-specific data operations and training and testing activities. This dataset is a sample of processed NuRec reconstructions to test SFT, Eval, and RL workflows.

The scenes in this dataset are generated by and can be rendered by using [NVIDIA NuRec](https://www.nvidia.com/en-us/glossary/3d-reconstruction/). CARLA users can also utilize this dataset by leveraging the [NVIDIA NuRec integration in CARLA](https://carla.org/2025/06/11/release-0.9.16-pre/).

## Developer Tooling

Please see [https://huggingface.co/docs/huggingface_hub/v1.0.0.rc5/en/guides/download](https://huggingface.co/docs/huggingface_hub/v1.0.0.rc5/en/guides/download) for complete documentation on how to download dataset files. Example scripts are also included at the bottom of this document.

The NCore based reconstruction workflow that is the basis for downstream SFT or RL tasks can be found here:

- [Aplamayo Reconstruction Workflow](https://docs.nvidia.com/nurec/)

## Version History

| Version | Notes |
|:-----------------|:--------------------------------------------------|
| 26.02 | <ul><li>NuRec scenes have been constructed with NuRec 26.01 version.</li><li>Reduced processed count to 918</li></ul> |
| 25.07 | <ul><li>NuRec scenes have been constructed with NuRec 25.05</li><li>Added labels.json file for helping users who want to search by types of scenes based on: behavior, layout, lighting, road types, surface conditions, traffic density, vrus presence, and weather. (Note this is only available for files under Batch0002 and onwards)</li><li>A front camera video file for each clips so that users can assess the scene before opening the usdz. (Note this is only available for files under Batch0002 and onwards)</li></ul> |
| 25.05 | NuRec scenes have been constructed with NuRec 25.0 |

## License/Terms of Use
[NVIDIA Autonomous Vehicle Dataset License Agreement](https://huggingface.co/datasets/nvidia/PhysicalAI-Autonomous-Vehicles/blob/main/LICENSE.pdf)

## Dataset Owner(s)
NVIDIA Corporation

## Ethical Considerations
NVIDIA believes Trustworthy AI is a shared responsibility and we have established policies and practices to enable development for a
wide array of AI applications. When downloaded or used in accordance with our terms of service, developers should work with their
internal model team to ensure this model meets requirements for the relevant industry and use case and addresses unforeseen product misuse.

Please report security vulnerabilities or NVIDIA AI Concerns [here](https://www.nvidia.com/en-us/support/submit-security-vulnerability/).

## DATASET

### Dataset Format
The scenes are stored by batches containing a number of clip folders listed by their UUIDs. Each UUID folder contains:
* usdz file (always)
* labels.json file (in most cases)
* camera_front_wide_120fov.mp4 (in most cases)

Each reconstructed scene is stored as a USDZ File containing the following:

| Files | Description |
| ----- | ----- |
| checkpoint.ckpt | Trained neural network weights |
| data\_info.json | Timestamp and frame range detail per sensors |
| datasource\_summary.json| Sensor track and poses summary |
| default.usda| Main scene file referencing all assets and configurations |
| dome\_light.usda | Describe dome lighting for scene illumination |
| map.xodr | OpenDRIVE map file |
| mesh.ply | Polygon mesh file for 3D geometry |
| mesh.usd | USD file for 3D mesh |
| mesh\_ground.ply | Polygon mesh file for ground surface geometry |
| mesh\_ground.usd | USD file for ground mesh |
| metadata.yaml | YAML file with scene metadata |
| parsed\_config.yaml | YAML configuration file |
| rig\_trajectories.json | JSON file containing sensor rig trajectory data |
| rig\_trajectories.usda | Rig trajectories in the USD scene |
| sequence\_tracks.json | JSON file with object tracking information |
| sequence\_tracks.usda | Object sequence tracks in the USD scene |
| volume.nurec | volumetric data file for neural reconstruction|
| volume.usda | USD ASCII file describing volumetric data in the scene |

The labels.json contains the following fields where each field except VRUs can have multiple values:
* Behavior types: {driving_straight, stop, left_lane_change, right_lane_change, right_turn, left_turn, unspecified, reverse}
* Layout types: {straight_road, intersection, underpass, unspecified, bridge, construction_zone, parking_lot, pedestrian_crossing, ramp, roundabout, railway_crossing}
* Road types: {residential, highways, urban, unspecified, rural, other}
* Weather types: {clear/cloudy, unspecified, rain, fog}
* Surface conditions: {dry, unspecified, wet}
* Lighting types: {daytime, unspecified, nighttime}
* VRUs: {True, False}
* Traffic density: {low, medium, high, unspecified}

## Downloading Data

Please see https://huggingface.co/docs/huggingface_hub/v1.0.0.rc5/en/guides/download for complete documentation on how
download dataset files. The code below is just an example only.
```python
from huggingface_hub import login, snapshot_download

def main():
hf_api_token = os.getenv("HF_TOKEN")

login(token=hf_api_token)

# Download an entire repository
snapshot_download(repo_id="nvidia/PhysicalAI-Autonomous-Vehicles-NuRec", repo_type="dataset")

# Download all the files in a folder
snapshot_download(repo_id="nvidia/PhysicalAI-Autonomous-Vehicles-NuRec", repo_type="dataset", allow_patterns="sample_set/25.07_release/Batch0002/001b28cb-b8f7-4627-ae65-fda88612d5bf/*")

# Download an individual file
snapshot_download(repo_id="nvidia/PhysicalAI-Autonomous-Vehicles-NuRec", repo_type="dataset", allow_patterns="sample_set/25.07_release/Batch0002/001b28cb-b8f7-4627-ae65-fda88612d5bf/001b28cb-b8f7-4627-ae65-fda88612d5bf.usdz")

if __name__ == "__main__":
main()
```

Downloading usdz based upon categories in labels.json
```python
import argparse
from pathlib import Path
import json
import os

from huggingface_hub import login, snapshot_download

def string_to_boolean(s):
s = s.strip().lower() # Normalize the string
if s in ('true', '1', 'yes', 'on'):
return True

return False

def main():
valid_categories = ["behavior", "layout", "lighting", "road_types", "surface_conditions", "traffic_density", "vrus", "weather"]

parser = argparse.ArgumentParser(
description="Downloads usdz clips based upon criteria specified in the labels.json"
)
parser.add_argument(
"--local-dir", type=str, required=True, help="The path to store the usdz"
)
parser.add_argument(
"--category",
type=str,
required=True,
choices=valid_categories,
help="The specified category in the labels.json. Must be one of: %(choices)",
)
parser.add_argument(
"--value",
type=str,
required=True,
help="The specified value in the category",
)

args = parser.parse_args()

hf_api_token = os.getenv("HF_TOKEN")

login(token=hf_api_token)

# First download all the labels.json files
print(f"Downloading dataset labels.json to {args.local_dir}.")

snapshot_download(repo_id="nvidia/PhysicalAI-Autonomous-Vehicles-NuRec", repo_type="dataset", allow_patterns="*.json", local_dir=args.local_dir)

category = args.category
value = args.value

# Find all of the labels.json files that have been downloaded
local_dir = Path(args.local_dir)
label_paths = local_dir.rglob("labels.json")

# Filter through the labels.json and find all usdz that match our criteria
paths_to_download = {}

print(f"Filtering usdz downloads based upon labels.json downloaded with criteria {category} and {value}.")

for label_path in label_paths:
with open(label_path, "r", encoding="utf-8") as f:
metadata = json.load(f)

if category in metadata:
if category == "vrus":
if string_to_boolean(value) == metadata["vrus"]:
paths_to_download[label_path.parent] = True
else:
if value in metadata[args.category]:
paths_to_download[label_path.parent] = True

print(f"Found {len(paths_to_download)} that matched criteria.")

# Download the selected usdz and front camera mp4
for path in paths_to_download.keys():
relative_path = path.relative_to(local_dir)

print(f"Downloading usdz and front camera at path {relative_path}")
snapshot_download(repo_id="nvidia/PhysicalAI-Autonomous-Vehicles-NuRec", repo_type="dataset", allow_patterns=f"{relative_path}/*", local_dir=args.local_dir)

if __name__ == "__main__":
main()
```
## Reference(s):
@article{wu20253dgut,
title={3DGUT: Enabling Distorted Cameras and Secondary Rays in Gaussian Splatting},
author={Wu, Qi and Martinez Esturo, Janick and Mirzaei, Ashkan and Moenne-Loccoz, Nicolas and Gojcic, Zan},
journal={Conference on Computer Vision and Pattern Recognition (CVPR)},
year={2025}
}

任务类别：
- 机器人学
标签：
- physicalAI
---
## 🚀 最新动态（2025年10月22日，新增大量场景并优化易用性）！！
目前我们已完成以下更新：
* 将**NuRec场景数量提升至924个**！
* 新增`labels.json`文件，方便用户根据以下维度**按场景类型检索**：行为、布局、光照、道路类型、路面状况、交通密度、VRUs（Vulnerable Road Users，易受伤害道路使用者）存在情况、天气。（注：该文件仅在Batch0002及后续批次中可用）
* 为每个片段新增**前置摄像头视频文件**，方便用户在打开usdz（USDZ）文件前预览场景。（注：该文件仅在Batch0002及后续批次中可用）

可在`sample_set/25.07_release`文件夹中查找900余个场景。

## 数据集说明
本数据集为神经重建数据集，包含经3D重建的驾驶场景。单场景时长约20秒，以usdz（USDZ）文件格式存储，同时附带对应的xodr（OpenDRIVE）地图文件与表面网格。该重建结果通过6个摄像头视角生成：前广角120°、前长焦30°、左右交叉视角120°以及左右后视角70°。用户可利用这些3D重建的驾驶场景训练、测试自动驾驶（Autonomous Vehicle, AV）系统。本数据集仅可用于商用或非商用的自动驾驶相关用途。
<!-- <video controls autoplay src="https://cdn-uploads.huggingface.co/production/uploads/671aac6b5f7cde5ac3e82f26/QChnbHmjDi6aS5rXG3_uJ.mp4"></video> -->
<!-- <video controls autoplay src="https://cdn-uploads.huggingface.co/production/uploads/671aac6b5f7cde5ac3e82f26/TJDotgIksh5YOPFDzzcX8.qt"></video> -->
<video controls autoplay src="https://cdn-uploads.huggingface.co/production/uploads/671aac6b5f7cde5ac3e82f26/Kv_kpnk9hqATRNxPQJeL4.qt"></video>

## 数据集所有者
NVIDIA公司

## 数据集创建日期
2025年6月9日

## 使用许可
[NVIDIA自动驾驶数据集许可协议](https://huggingface.co/datasets/nvidia/PhysicalAI-Autonomous-Vehicles-NuRec/blob/main/License.pdf)

## 预期用途
本数据集旨在让自动驾驶开发者体验NuRec的能力并试用3DGUT。用户可利用本数据集开展自动驾驶系统的各类测试与实验，以及训练基于摄像头数据与重建数据的AI模型。本数据集的场景由[NVIDIA NuRec](https://www.nvidia.com/en-us/glossary/3d-reconstruction/)生成，也可通过该工具渲染。CARLA用户可借助[CARLA中集成的NVIDIA NuRec功能](https://carla.org/2025/06/11/release-0.9.16-pre/)使用本数据集。

## 数据集特征
### 数据采集方式
* [自动/传感器] - [机器生成]
### 标注方式
* [自动/传感器] - [机器生成]

## 数据集格式
场景按批次存储，每个批次包含若干以UUID命名的片段文件夹。每个UUID文件夹包含：
* 始终存在usdz文件
* 多数情况下包含`labels.json`文件
* 多数情况下包含`camera_front_wide_120fov.mp4`文件

每个重建场景的USDZ文件中包含以下内容：
| 文件名称 | 描述 |
| ----- | ----- |
| checkpoint.ckpt | 训练好的神经网络权重文件 |
| data_info.json | 包含各传感器的时间戳与帧范围详情 |
| datasource_summary.json | 传感器轨迹与位姿汇总文件 |
| default.usda | 引用所有资产与配置的主场景文件 |
| dome_light.usda | 用于描述场景照明的穹顶光照文件 |
| map.xodr | OpenDRIVE地图文件 |
| mesh.ply | 用于3D几何的多边形网格文件 |
| mesh.usd | 用于3D网格的USD文件 |
| mesh_ground.ply | 用于地面几何的多边形网格文件 |
| mesh_ground.usd | 地面网格的USD文件 |
| metadata.yaml | 包含场景元数据的YAML文件 |
| parsed_config.yaml | YAML配置文件 |
| rig_trajectories.json | 包含传感器套件轨迹数据的JSON文件 |
| rig_trajectories.usda | USD场景中的传感器套件轨迹文件 |
| sequence_tracks.json | 包含目标跟踪信息的JSON文件 |
| sequence_tracks.usda | USD场景中的目标序列轨迹文件 |
| volume.nurec | 用于神经重建的体数据文件 |
| volume.usda | 描述场景中体数据的USD ASCII文件 |

`labels.json`包含以下字段，除VRUs外，其余字段均可包含多个取值：
* 行为类型：{driving_straight, stop, left_lane_change, right_lane_change, right_turn, left_turn, unspecified, reverse} → 直行、停车、左变道、右变道、右转、左转、未指定、倒车
* 布局类型：{straight_road, intersection, underpass, unspecified, bridge, construction_zone, parking_lot, pedestrian_crossing, ramp, roundabout, railway_crossing} → 直线路段、交叉口、地下通道、未指定、桥梁、施工区域、停车场、人行横道、坡道、环岛、铁路道口
* 道路类型：{residential, highways, urban, unspecified, rural, other} → 居民区道路、高速公路、城市道路、未指定、乡村道路、其他
* 天气类型：{clear/cloudy, unspecified, rain, fog} → 晴/多云、未指定、降雨、雾天
* 路面状况：{dry, unspecified, wet} → 干燥、未指定、湿润
* 光照类型：{daytime, unspecified, nighttime} → 白天、未指定、夜间
* VRUs：{True, False} → 存在、不存在
* 交通密度：{low, medium, high, unspecified} → 低、中、高、未指定

## 数据集量化
样本数量：900余个usdz文件（后续将新增更多）
总存储容量：约4.5 TB

## 数据下载
完整的数据集下载文档请参阅：https://huggingface.co/docs/huggingface_hub/v1.0.0.rc5/en/guides/download，以下代码仅为示例。

python
from huggingface_hub import login, snapshot_download

def main():
hf_api_token = os.getenv("HF_TOKEN")

login(token=hf_api_token)

# Download an entire repository
snapshot_download(repo_id="nvidia/PhysicalAI-Autonomous-Vehicles-NuRec", repo_type="dataset")

# Download all the files in a folder
snapshot_download(repo_id="nvidia/PhysicalAI-Autonomous-Vehicles-NuRec", repo_type="dataset", allow_patterns="sample_set/25.07_release/Batch0002/001b28cb-b8f7-4627-ae65-fda88612d5bf/*")

# Download an individual file
snapshot_download(repo_id="nvidia/PhysicalAI-Autonomous-Vehicles-NuRec", repo_type="dataset", allow_patterns="sample_set/25.07_release/Batch0002/001b28cb-b8f7-4627-ae65-fda88612d5bf/001b28cb-b8f7-4627-ae65-fda88612d5bf.usdz")

if __name__ == "__main__":
main()

根据`labels.json`中的分类下载usdz文件：
python
import argparse
from pathlib import Path
import json
import os

from huggingface_hub import login, snapshot_download

def string_to_boolean(s):
s = s.strip().lower() # Normalize the string
if s in ('true', '1', 'yes', 'on'):
return True

return False

def main():
valid_categories = ["behavior", "layout", "lighting", "road_types", "surface_conditions", "traffic_density", "vrus", "weather"]

parser = argparse.ArgumentParser(
description="Downloads usdz clips based upon criteria specified in the labels.json"
)
parser.add_argument(
"--local-dir", type=str, required=True, help="The path to store the usdz"
)
parser.add_argument(
"--category",
type=str,
required=True,
choices=valid_categories,
help="The specified category in the labels.json. Must be one of: %(choices)",
)
parser.add_argument(
"--value",
type=str,
required=True,
help="The specified value in the category",
)

args = parser.parse_args()

hf_api_token = os.getenv("HF_TOKEN")

login(token=hf_api_token)

# First download all the labels.json files
print(f"Downloading dataset labels.json to {args.local_dir}.")

snapshot_download(repo_id="nvidia/PhysicalAI-Autonomous-Vehicles-NuRec", repo_type="dataset", allow_patterns="*.json", local_dir=args.local_dir)

category = args.category
value = args.value

# Find all of the labels.json files that have been downloaded
local_dir = Path(args.local_dir)
label_paths = local_dir.rglob("labels.json")

# Filter through the labels.json and find all usdz that match our criteria
paths_to_download = {}

print(f"Filtering usdz downloads based upon labels.json downloaded with criteria {category} and {value}.")

for label_path in label_paths:
with open(label_path, "r", encoding="utf-8") as f:
metadata = json.load(f)

if category in metadata:
if category == "vrus":
if string_to_boolean(value) == metadata["vrus"]:
paths_to_download[label_path.parent] = True
else:
if value in metadata[args.category]:
paths_to_download[label_path.parent] = True

print(f"Found {len(paths_to_download)} that matched criteria.")

# Download the selected usdz and front camera mp4
for path in paths_to_download.keys():
relative_path = path.relative_to(local_dir)

print(f"Downloading usdz and front camera at path {relative_path}")
snapshot_download(repo_id="nvidia/PhysicalAI-Autonomous-Vehicles-NuRec", repo_type="dataset", allow_patterns=f"{relative_path}/*", local_dir=args.local_dir)

if __name__ == "__main__":
main()

## 参考文献
bibtex
@article{wu20253dgut,
title={3DGUT: Enabling Distorted Cameras and Secondary Rays in Gaussian Splatting},
author={Wu, Qi and Martinez Esturo, Janick and Mirzaei, Ashkan and Moenne-Loccoz, Nicolas and Gojcic, Zan},
journal={Conference on Computer Vision and Pattern Recognition (CVPR)},
year={2025}
}

## 伦理考量
NVIDIA认为可信AI是一项共同责任，我们已制定相关政策与实践规范，以支持各类AI应用的开发。开发者在按照本服务条款下载或使用本数据集时，应与其内部模型团队协作，确保该模型符合相关行业与应用场景的要求，并防范未预见的产品误用。

请通过[此链接](https://www.nvidia.com/en-us/support/submit-security-vulnerability/)报告安全漏洞或NVIDIA AI相关问题。

搜集汇总

背景与挑战

背景概述

该数据集是一个包含900多个3D重建驾驶场景的神经重建数据集，适用于自动驾驶系统的训练和测试。每个场景以USDZ文件格式存储，附带多种辅助文件，如地图、网格和轨迹数据，支持多种场景分类和搜索功能。