Rexverse-2M-Camera

# Rexverse-2M-Camera  Per-image camera parameter annotations for the **Rexverse-2M** dataset (the released image subset of [IDEA-Research/Rexverse-2M](https://huggingface.co/datasets/IDEA-Research/Rexverse-2M); 415,857 images across 84 shards), captioned by the [**Puffin-World**](https://huggingface.co/ACERobotics/Puffin-World) model. More captioned datasets are provided in our [**Puffin-16M**](https://kangliao929.github.io/projects/puffin-16m/) website. The collage above visualizes the camera maps on sample images — each pair shows the **up field** (green arrows: the projected gravity-up direction) and the **latitude field** (colored contours: angle above/below the horizon). ## Format One `.tar` per shard (`rexverse_0000.tar` … ), each containing one `.json` per image whose name matches the source image stem. Each JSON holds the predicted monocular camera parameters: | Field | Meaning | Unit | |-------|---------|------| | `roll` | camera roll | radians | | `pitch` | camera pitch | radians | | `vfov` | vertical field-of-view | radians | | `k1` | radial distortion coefficient | – | | `parse_ok` | whether the model output parsed within valid ranges | bool | Example: ```json {"roll": 0.0123, "pitch": -0.0871, "vfov": 1.0123, "k1": 0.0000, "parse_ok": true} ``` ## Camera Parameter Distributions Histograms of the predicted roll / pitch / vertical-FoV over the whole dataset (proportion of valid samples per 10° bin; `parse_ok=False` excluded).  | split | roll μ / med / σ | pitch μ / med / σ | FoV μ / med / σ | |-------|------------------|-------------------|-----------------| | all (414,634) | 0.2° / 0.0° / 6.8° | −4.0° / −0.9° / 12.9° | 32.2° / 28.9° / 11.1° | - **Roll** is sharply peaked at 0° (web images shot upright/level). - **Pitch** is centered slightly negative with a broad spread (σ ≈ 12.9°). - **FoV** spans a wide range (σ ≈ 11.1°, median ≈ 29°) — consistent with the diverse focal lengths of large-scale web imagery. If you'd like a dataset with a more diverse and uniform distribution of camera parameters, please refer to our [Puffin-4M](https://huggingface.co/datasets/KangLiao/Puffin-4M) and [Puffin-16M](https://huggingface.co/datasets/KangLiao/Puffin-16M) datasets. ### Dataset Download You can download the entire dataset using the following command: ```bash hf download KangLiao/Rexverse-2M-Camera --repo-type dataset ``` ### From Camera Parameters to Up and Latitude Fields The released `(roll, pitch, vfov, k1)` annotations can be converted into the dense perspective-field representation used by Puffin-World. The conversion computes focal length from `vfov`, constructs a radial camera, and maps roll and pitch to the gravity direction. [`get_perspective_field`](https://github.com/KangLiao929/Puffin/blob/main/Puffin-World/scripts/camera/geometry/perspective_fields.py) then returns a normalized 2-channel **up field** and a 1-channel **latitude field**. The up field encodes the projected world-up direction at every pixel, while the latitude field measures each viewing ray's angular elevation relative to the horizon. ```python import json import torch from scripts.camera.geometry.camera import SimpleRadial from scripts.camera.geometry.gravity import Gravity from scripts.camera.geometry.perspective_fields import get_perspective_field from scripts.camera.utils.conversions import fov2focal with open("camera.json") as f: annotation = json.load(f) roll, pitch, vfov, k1 = ( annotation[key] for key in ("roll", "pitch", "vfov", "k1") ) H, W = 512, 512 f = float(fov2focal(torch.tensor(vfov), H)) camera = SimpleRadial(torch.tensor( [W, H, f, f, W / 2, H / 2, k1, 0.0] ).float()).scale(torch.tensor([1.0, 1.0])) gravity = Gravity.from_rp(torch.tensor(roll), torch.tensor(pitch)) up_field, latitude_field = get_perspective_field(camera, gravity) # Shapes: [1, 2, H, W] and [1, 1, H, W] ``` Run the example from the `Puffin-World` directory. Use [`plot_vector_fields` and `plot_latitudes`](https://github.com/KangLiao929/Puffin/blob/main/Puffin-World/scripts/camera/visualization/viz2d.py) to render up-field arrows and latitude heatmaps or contours. See [`save_pf_visualization`](https://github.com/KangLiao929/Puffin/blob/main/Puffin-World/scripts/demo/physics_perception.py) for an end-to-end visualization example. ### Caption Pipeline Beyond this captioned dataset, we also release **a complete captioning pipeline** for annotating camera parameters for arbitrary datasets, analyzing camera parameter distributions, and visualizing the corresponding camera maps. The pipeline is available in our [GitHub repository](https://github.com/KangLiao929/Puffin/blob/main/Puffin-World/documents/ANNOTATION_CAMERA.md). ### Citation If you find the captioned dataset useful for your research or applications, please cite the following papers using these BibTeX entries: ```bibtex @article{liao2025puffin, title={Thinking with Camera: A Unified Multimodal Model for Camera-Centric Understanding and Generation}, author={Liao, Kang and Wu, Size and Wu, Zhonghua and Jin, Linyi and Wang, Chao and Wang, Yikai and Wang, Fei and Li, Wei and Loy, Chen Change}, journal={arXiv preprint arXiv:2510.08673}, year={2025} } @article{liao2026puffinworld, title={Puffin-World: Scaling a Unified Multimodal Model with Native 3D World States}, author={Liao, Kang and Luo, Yihang and Wu, Xiao-Ming and Jin, Linyi and Wu, Size and Lin, Chunyu and Zhao, Yao and Wang, Fei and Li, Wei and Loy, Chen Change}, journal={arXiv preprint arXiv:2609.04196}, year={2026} } ```