objaverse-xl

# Objaverse-XL

<a href="//arxiv.org/abs/2307.05663" target="_blank">
<img src="https://img.shields.io/badge/arXiv-2307.05663-<COLOR>">
</a>

Objaverse-XL is an open dataset of over 10 million 3D objects!

With it, we train Zero123-XL, a foundation model for 3D, observing incredible 3D generalization abilities: 🧵👇

<img src="https://mattdeitke.com/static/1cdcdb2ef7033e177ca9ae2975a9b451/9c1ca/objaverse-xl.webp">

## Scale Comparison

Objaverse 1.0 was released back in December. It was a step in the right direction, but still relatively small with 800K objects.

Objaverse-XL is over an order of magnitude larger and much more diverse!

<img src="https://github.com/allenai/objaverse-rendering/assets/28768645/43833dd3-ec97-4a3d-8782-00a6aea584b4">

## Unlocking Generalization

Compared to the original Zero123 model, Zero123-XL improves remarkably in 0-shot generalization abilities, even being able to perform novel view synthesis on sketches, cartoons, and people!

A ton more examples in the [📝 paper](https://arxiv.org/abs/2307.05663) :)

<img src="https://github.com/allenai/objaverse-rendering/assets/28768645/8470e4df-e39d-444b-9871-58fbee4b87fd">

## Image → 3D

With the base Zero123-XL foundation model, we can perform image → 3D using [DreamFusion](https://dreamfusion3d.github.io/), having the model guide a NeRF to generate novel views!

<video autoplay muted loop controls>
<source src="https://github.com/allenai/objaverse-rendering/assets/28768645/571852cd-dc02-46ce-b2bb-88f64a67d0ac" type="video/mp4">
</video>

## Text → 3D

Text-to-3D comes for free with text → image models, such as with SDXL here, providing the initial image!

<video autoplay muted loop controls>
<source src="https://github.com/allenai/objaverse-rendering/assets/28768645/96255b42-8158-4c7a-8308-7b0f1257ada8" type="video/mp4">
</video>

## Scaling Trends

Beyond that, we show strong scaling trends for both Zero123-XL and [PixelNeRF](https://alexyu.net/pixelnerf/)!

<img src="https://github.com/allenai/objaverse-rendering/assets/28768645/0c8bb433-27df-43a1-8cb8-1772007c0899">

## Tutorial

Check out the [Google Colab tutorial](https://colab.research.google.com/drive/15XpZMjrHXuky0IgBbXcsUtb_0g-XWYmN?usp=sharing) to download Objaverse-XL.

Polycam data is available by Polycam to academic researchers for non-commercial use upon request and approval from Polycam. For access please fill out [this form](https://forms.gle/HUjYVtS9GKVS5QBXA).

## License

The use of the dataset as a whole is licensed under the ODC-By v1.0 license. Individual objects in Objaverse-XL are licensed under different licenses.

## Citation

To cite Objaverse-XL, please cite our [📝 arXiv](https://arxiv.org/abs/2307.05663) paper with the following BibTeX entry:

```bibtex
@article{objaverseXL,
title={Objaverse-XL: A Universe of 10M+ 3D Objects},
author={Matt Deitke and Ruoshi Liu and Matthew Wallingford and Huong Ngo and
Oscar Michel and Aditya Kusupati and Alan Fan and Christian Laforte and
Vikram Voleti and Samir Yitzhak Gadre and Eli VanderBilt and
Aniruddha Kembhavi and Carl Vondrick and Georgia Gkioxari and
Kiana Ehsani and Ludwig Schmidt and Ali Farhadi},
journal={arXiv preprint arXiv:2307.05663},
year={2023}
}
```

Objaverse 1.0 is available on 🤗Hugging Face at [@allenai/objaverse](https://huggingface.co/datasets/allenai/objaverse). To cite it, use:

```bibtex
@article{objaverse,
title={Objaverse: A Universe of Annotated 3D Objects},
author={Matt Deitke and Dustin Schwenk and Jordi Salvador and Luca Weihs and
Oscar Michel and Eli VanderBilt and Ludwig Schmidt and
Kiana Ehsani and Aniruddha Kembhavi and Ali Farhadi},
journal={arXiv preprint arXiv:2212.08051},
year={2022}
}
```

# Objaverse-XL
<a href="//arxiv.org/abs/2307.05663" target="_blank">
<img src="https://img.shields.io/badge/arXiv-2307.05663-<COLOR>">
</a>

Objaverse-XL 是一款包含超1000万个3D物体的开源数据集！

依托该数据集，我们训练得到了面向3D任务的基础模型Zero123-XL，其展现出了令人惊艳的3D泛化能力：🧵👇

<img src="https://mattdeitke.com/static/1cdcdb2ef7033e177ca9ae2975a9b451/9c1ca/objaverse-xl.webp">

## 规模对比

Objaverse 1.0 于2022年12月发布，虽迈出了正确的研发方向，但规模仍相对有限，仅包含80万个3D物体。

Objaverse-XL 的规模比前者高出一个数量级以上，且数据多样性也得到了大幅提升！

<img src="https://github.com/allenai/objaverse-rendering/assets/28768645/43833dd3-ec97-4a3d-8782-00a6aea584b4">

## 解锁泛化能力

相较于初代Zero123模型，Zero123-XL在零样本（zero-shot）泛化能力上实现了显著提升，甚至能够对素描、卡通形象以及人物进行新颖视角合成。

更多示例可参阅[📝 论文](https://arxiv.org/abs/2307.05663) :)

<img src="https://github.com/allenai/objaverse-rendering/assets/28768645/8470e4df-e39d-444b-9871-58fbee4b87fd">

## 图像→3D

依托基础模型Zero123-XL，我们可以借助[DreamFusion](https://dreamfusion3d.github.io/)实现图像到3D的转换，通过该模型引导神经辐射场（NeRF）生成新颖视角视图！

<video autoplay muted loop controls>
<source src="https://github.com/allenai/objaverse-rendering/assets/28768645/571852cd-dc02-46ce-b2bb-88f64a67d0ac" type="video/mp4">
</video>

## 文本→3D

借助文本到图像模型（例如此处使用的Stable Diffusion XL（SDXL））生成初始图像，即可免费实现文本到3D的转换！

<video autoplay muted loop controls>
<source src="https://github.com/allenai/objaverse-rendering/assets/28768645/96255b42-8158-4c7a-8308-7b0f1257ada8" type="video/mp4">
</video>

## 缩放趋势

除此之外，我们还验证了Zero123-XL与[PixelNeRF](https://alexyu.net/pixelnerf/)均具备良好的缩放趋势！

<img src="https://github.com/allenai/objaverse-rendering/assets/28768645/0c8bb433-27df-43a1-8cb8-1772007c0899">

## 使用教程

可参阅[Google Colab使用教程](https://colab.research.google.com/drive/15XpZMjrHXuky0IgBbXcsUtb_0g-XWYmN?usp=sharing)下载Objaverse-XL数据集。

Polycam提供的数据集可由Polycam向学术研究人员开放，仅可用于非商业用途，需经Polycam申请并获得批准。如需获取权限，请填写[此申请表单](https://forms.gle/HUjYVtS9GKVS5QBXA)。

## 授权协议

数据集整体采用ODC-By v1.0协议进行授权，而Objaverse-XL中的单个3D物体则采用不同的授权协议。

## 引用方式

如需引用Objaverse-XL，请通过以下BibTeX条目引用我们的[📝 arXiv论文](https://arxiv.org/abs/2307.05663)：

bibtex
@article{objaverseXL,
title={"Objaverse-XL: A Universe of 10M+ 3D Objects"},
author={Matt Deitke and Ruoshi Liu and Matthew Wallingford and Huong Ngo and
Oscar Michel and Aditya Kusupati and Alan Fan and Christian Laforte and
Vikram Voleti and Samir Yitzhak Gadre and Eli VanderBilt and
Aniruddha Kembhavi and Carl Vondrick and Georgia Gkioxari and
Kiana Ehsani and Ludwig Schmidt and Ali Farhadi},
journal={arXiv preprint arXiv:2307.05663},
year={2023}
}

Objaverse 1.0 可在🤗 Hugging Face上的[@allenai/objaverse](https://huggingface.co/datasets/allenai/objaverse)获取。其引用方式如下：

bibtex
@article{objaverse,
title={"Objaverse: A Universe of Annotated 3D Objects"},
author={Matt Deitke and Dustin Schwenk and Jordi Salvador and Luca Weihs and
Oscar Michel and Eli VanderBilt and Ludwig Schmidt and
Kiana Ehsani and Aniruddha Kembhavi and Ali Farhadi},
journal={arXiv preprint arXiv:2212.08051},
year={2022}
}