PhysicalAI-Autonomous-Vehicles-Foundation4D

# PhysicalAI Autonomous Vehicles - Foundation4D ## Dataset Description: This release extends NVIDIA’s existing public [PhysicalAI-Autonomous-Vehicles dataset](https://huggingface.co/datasets/nvidia/PhysicalAI-Autonomous-Vehicles) with additional labels for 1253 specific clips, including: - Metric depth - Per-voxel occupancy, semantic labels, and velocity These labels are intended to provide target values and labels that can be used to validate NVIDIA Foundation4D and other autonomous vehicle perception models as well as autolabeling tools for physical AI. This dataset is for non-commercial uses only. ## Dataset Owner(s): NVIDIA Corporation ## Dataset Creation Date: October 2026 ## Version: v1.0 Previous Version(s): None ## License/Terms of Use: https://huggingface.co/datasets/nvidia/PhysicalAI-Autonomous-Vehicles/blob/main/LICENSE.pdf ## Intended Usage: This dataset is intended to support validation, testing, and replication for autonomous vehicle perception models as well as autolabeling tools. Users can use these ground-truth labels to benchmark and validate their own model outputs on the [PhysicalAI-Autonomous-Vehicles dataset](https://huggingface.co/datasets/nvidia/PhysicalAI-Autonomous-Vehicles). ## Dataset Characterization **Data Collection Method** The clips were collected by real-world vehicle fleets. The ground-truth labels were generated from temporally aggregated LiDAR measurements, and machine-generated labels, including 2D semantic labels, 2D instance masks, and 3D cuboid labels. **Labeling Method** Automated ## Dataset Format Format: NCore aux data stored as Zarr v2 in indexed TAR (`.itar`) archives. **Metric depth** ``` {clip}.aux.depth.zarr.itar └── aux/ └── depth/ ├── {camera_id}/ │ ├── {timestamp_us} float16[H, W] │ │ # Radial camera range in metres │ │ # 0 = unavailable/excluded depth │ └── {next_timestamp_us} float16[H, W] └── {next_camera_id}/ ├── {timestamp_us} float16[H, W] └── {next_timestamp_us} float16[H, W] # Rendered radial range in metres, using rolling-shutter camera geometry. # 0 = no retained valid depth, including unavailable/excluded pixels. # Keys are each camera's frame-END timestamps in microseconds. # H and W are image height and width, respectively. # H, W, timestamps, and frame counts are camera-specific. ``` **Per-voxel occupancy, semantic labels, and velocity** ``` {clip}.aux.occupancy.zarr.itar └── aux/ └── occupancy/ └── lidar_virtual/ ├── {timestamp_us}/ │ ├── voxel_coords int16[N, 3] │ │ # Zero-based XYZ grid indices │ ├── semantic_labels uint8[N] │ │ # Class IDs (detailed class set below) │ ├── velocity float32[N, 3] │ │ # Ground-relative cuboid translational velocity estimate, │ │ # expressed as XYZ components in current RIG axes, in m/s. │ │ # No ego velocity subtraction. │ │ # Static-source rows are zero-filled. │ ├── occupancy_probs float32[N] │ │ # 1.0 for each retained occupied voxel │ │ # Missing voxel ≠ certified free space │ ├── voxel_size float32[1] = [0.1] metres │ │ # Isotropic 10 cm edge length │ ├── grid_origin float32[3] = [-150, -100, -3] metres │ │ # Lower XYZ corner in RIG │ └── grid_shape int32[3] = [3000, 2000, 80] │ # Voxel counts in XYZ order └── {next_timestamp_us}/ ├── voxel_coords int16[N_next, 3] ├── semantic_labels uint8[N_next] ├── velocity float32[N_next, 3] ├── occupancy_probs float32[N_next] ├── voxel_size float32[1] ├── grid_origin float32[3] └── grid_shape int32[3] # Timestamps: camera_front_wide_120fov frame-END time, microseconds. # Occupied voxel count N may vary between frames. # All per-voxel arrays align by row within each frame. # Grid parameters are constant per clip and repeated per frame. # Voxel centre (RIG metres) = grid_origin + (voxel_coords + 0.5) × voxel_size. ``` Details of semantic labels: ``` { 0: "road", 1: "sidewalk", 2: "building", 3: "wall", 4: "fence", 5: "pole", 6: "traffic light", 7: "traffic sign", 8: "vegetation", 9: "terrain", 11: "person", 12: "rider", 13: "car", 14: "truck", 15: "bus", 16: "train", 17: "motorcycle", 18: "bicycle", 20: "unknown", } Exclusion: { 10: "sky", 19: "egocar", } ``` ## Dataset Quantification Record Count: 1253 clips Feature Count: 2 label types (metric depth; per-voxel occupancy, semantic labels, and velocity) ## Reference(s): NVIDIA used Meta’s Segment Anything Model 2 (SAM 2), made available under the [Apache 2.0 License](https://github.com/facebookresearch/sam2/blob/main/LICENSE), as an auxiliary tool to generate intermediate segmentation masks during dataset preparation. These masks were processed through NVIDIA’s pipeline to produce portions of the released labels. ## Ethical Considerations: NVIDIA believes Trustworthy AI is a shared responsibility and we have established policies and practices to enable development for a wide array of AI applications. Developers should work with their internal developer teams to ensure this dataset meets requirements for the relevant industry and use case and addresses unforeseen product misuse. Please report quality, risk, security vulnerabilities or NVIDIA AI Concerns [here](https://www.nvidia.com/en-us/support/submit-security-vulnerability/).