ScanNet-Depth-DA3-Aligned

# ScanNet-Depth-DA3-Aligned Per-frame depth annotations for the ScanNet dataset, produced by **Depth-Anything-3 (DA3)** and then **aligned** to each scene's sparse depth from the original ScanNet reconstruction. We use this refined dataset for 3D world generation and reconstruction in our [**Puffin-World**](https://github.com/KangLiao929/Puffin). ## Sample Videos Each clip is a 1×3 comparison — **RGB | Original Depth | Our Aligned Depth** — with depth rendered by *vision banana* representation. It shows how the DA3-aligned depth (right) fills the holes and densifies the original sparse sensor depth (middle). <table> <tr><td width="50%"><video src="https://huggingface.co/datasets/KangLiao/ScanNet-Depth-DA3-Aligned/resolve/main/samples/sample_00.mp4" controls muted loop playsinline preload="metadata" width="100%"></video></td><td width="50%"><video src="https://huggingface.co/datasets/KangLiao/ScanNet-Depth-DA3-Aligned/resolve/main/samples/sample_01.mp4" controls muted loop playsinline preload="metadata" width="100%"></video></td></tr> <tr><td width="50%"><video src="https://huggingface.co/datasets/KangLiao/ScanNet-Depth-DA3-Aligned/resolve/main/samples/sample_02.mp4" controls muted loop playsinline preload="metadata" width="100%"></video></td><td width="50%"><video src="https://huggingface.co/datasets/KangLiao/ScanNet-Depth-DA3-Aligned/resolve/main/samples/sample_03.mp4" controls muted loop playsinline preload="metadata" width="100%"></video></td></tr> <tr><td width="50%"><video src="https://huggingface.co/datasets/KangLiao/ScanNet-Depth-DA3-Aligned/resolve/main/samples/sample_04.mp4" controls muted loop playsinline preload="metadata" width="100%"></video></td><td width="50%"><video src="https://huggingface.co/datasets/KangLiao/ScanNet-Depth-DA3-Aligned/resolve/main/samples/sample_05.mp4" controls muted loop playsinline preload="metadata" width="100%"></video></td></tr> <tr><td width="50%"><video src="https://huggingface.co/datasets/KangLiao/ScanNet-Depth-DA3-Aligned/resolve/main/samples/sample_06.mp4" controls muted loop playsinline preload="metadata" width="100%"></video></td><td width="50%"><video src="https://huggingface.co/datasets/KangLiao/ScanNet-Depth-DA3-Aligned/resolve/main/samples/sample_07.mp4" controls muted loop playsinline preload="metadata" width="100%"></video></td></tr> <tr><td width="50%"><video src="https://huggingface.co/datasets/KangLiao/ScanNet-Depth-DA3-Aligned/resolve/main/samples/sample_08.mp4" controls muted loop playsinline preload="metadata" width="100%"></video></td><td width="50%"><video src="https://huggingface.co/datasets/KangLiao/ScanNet-Depth-DA3-Aligned/resolve/main/samples/sample_09.mp4" controls muted loop playsinline preload="metadata" width="100%"></video></td></tr> <tr><td width="50%"><video src="https://huggingface.co/datasets/KangLiao/ScanNet-Depth-DA3-Aligned/resolve/main/samples/sample_10.mp4" controls muted loop playsinline preload="metadata" width="100%"></video></td><td width="50%"><video src="https://huggingface.co/datasets/KangLiao/ScanNet-Depth-DA3-Aligned/resolve/main/samples/sample_11.mp4" controls muted loop playsinline preload="metadata" width="100%"></video></td></tr> <tr><td width="50%"><video src="https://huggingface.co/datasets/KangLiao/ScanNet-Depth-DA3-Aligned/resolve/main/samples/sample_12.mp4" controls muted loop playsinline preload="metadata" width="100%"></video></td><td width="50%"><video src="https://huggingface.co/datasets/KangLiao/ScanNet-Depth-DA3-Aligned/resolve/main/samples/sample_13.mp4" controls muted loop playsinline preload="metadata" width="100%"></video></td></tr> <tr><td width="50%"><video src="https://huggingface.co/datasets/KangLiao/ScanNet-Depth-DA3-Aligned/resolve/main/samples/sample_14.mp4" controls muted loop playsinline preload="metadata" width="100%"></video></td><td width="50%"><video src="https://huggingface.co/datasets/KangLiao/ScanNet-Depth-DA3-Aligned/resolve/main/samples/sample_15.mp4" controls muted loop playsinline preload="metadata" width="100%"></video></td></tr> </table> ## Directory structure The archive mirrors the source ScanNet layout — one `.zip` per scene, under `scans/` — `scans/<scene_id>.zip` (e.g. `scans/scene0000_00.zip`). The scene ids match ScanNet and [ScanNet-Absolute-Camera](https://huggingface.co/datasets/KangLiao/ScanNet-Absolute-Camera), so depth pairs 1:1 with the source frames / absolute camera annotations. Each `<scene_id>.zip` unpacks to: ``` depth_da3/ ├── 00000.npy ├── 00001.npy ├── 00002.npy └── ... ``` Each `NNNNN.npy` is a float32 depth map — `np.load(...)` returns an array of shape `(H, W)`, one per source frame, indices matching the ScanNet frames. ## How the depth was produced - **Predicted** with Depth-Anything-3 (DA3). - **Aligned** to the sparse depth of the original ScanNet dataset (per-scene alignment against the sparse reconstruction), so each scene's DA3 depth is brought into a consistent, scale-aligned space. ## Usage ```python import numpy as np depth = np.load("depth_da3/00000.npy") # (H, W) float32 ``` ## Notes - ~1,468 scenes (`scans/`); each `.npy` frame ≈ 2 MB (float32), stored losslessly. - Companion camera annotations: [ScanNet-Absolute-Camera](https://huggingface.co/datasets/KangLiao/ScanNet-Absolute-Camera). ## Caption Pipeline Beyond the aligned dataset, we also release **a complete captioning pipeline** for annotating the dense depth map for arbitrary datasets, aligning with the sparse depth, and visualizing the corresponding depth maps. The pipeline is available in our [GitHub repository](https://github.com/KangLiao929/Puffin).