GaussianAnything: Interactive Point Cloud Latent Diffusion for 3D Generation

While 3D content generation has advanced significantly, existing methods still face challenges with input formats, latent space design, and output representations. This paper introduces a novel 3D generation framework that addresses these challenges, offering scalable, high-quality 3D generation with an interactive Point Cloud-structured Latent space. Our framework employs a Variational Autoencoder (VAE) with multi-view posed RGB-D(epth)-N(ormal) renderings as input, using a unique latent space design that preserves 3D shape information, and incorporates a cascaded latent diffusion model for improved shape-texture disentanglement. The proposed method, GaussianAnything, supports multi-modal conditional 3D generation, allowing for point cloud, caption, and single/multi-view image inputs. Notably, the newly proposed latent space naturally enables geometry-texture disentanglement, thus allowing 3D-aware editing. Experimental results demonstrate the effectiveness of our approach on multiple datasets, outperforming existing methods in both text- and image-conditioned 3D generation.

尽管三维内容生成技术已取得长足进展，但现有方法仍在输入格式、隐空间设计与输出表征层面面临诸多挑战。本文提出一款全新的三维生成框架以攻克上述难题，该框架依托交互式点云结构隐空间，可实现可扩展且高质量的三维内容生成。本框架采用以多视图带位姿的RGB-深度（Depth）-法线（Normal）渲染图为输入的变分自编码器（Variational Autoencoder，VAE），通过独特的隐空间设计保留三维形状信息，并集成级联隐扩散模型以优化形状与纹理的解耦效果。所提出的GaussianAnything方法支持多模态条件三维生成，可接收点云、文本描述以及单/多视图图像作为输入条件。尤为关键的是，本次提出的新型隐空间可自然实现几何与纹理的解耦，从而支持具备三维感知的编辑操作。实验结果表明，本方法在多个公开数据集上均验证了其有效性，在文本条件与图像条件的三维生成任务中均优于现有主流方法。