MeshAnything: Artist-Created Mesh Generation with Autoregressive Transformers

Recently, 3D assets created via reconstruction and generation have matched the quality of manually crafted assets, highlighting their potential for replacement. However, this potential is largely unrealized because these assets always need to be converted to meshes for 3D industry applications, and the meshes produced by current mesh extraction methods are significantly inferior to Artist-Created Meshes (AMs), i.e., meshes created by human artists. Specifically, current mesh extraction methods rely on dense faces and ignore geometric features, leading to inefficiencies, complicated post-processing, and lower representation quality. To address these issues, we introduce MeshAnything, a model that treats mesh extraction as a generation problem, producing AMs aligned with specified shapes. By converting 3D assets in any 3D representation into AMs, MeshAnything can be integrated with various 3D asset production methods, thereby enhancing their application across the 3D industry. The architecture of MeshAnything comprises a VQ-VAE and a shape-conditioned decoder-only transformer. We first learn a mesh vocabulary using the VQ-VAE, then train the shape-conditioned decoder-only transformer on this vocabulary for shape-conditioned autoregressive mesh generation. Our extensive experiments show that our method generates AMs with hundreds of times fewer faces, significantly improving storage, rendering, and simulation efficiencies, while achieving precision comparable to previous methods.

近年来，通过重建与生成技术制作的三维资产（3D assets）已达到手工创作资产的品质水准，凸显了其替代手工资产的潜力。然而，这一潜力尚未得到充分释放——这类资产往往需转换为网格（mesh）方可适配三维产业应用场景，而当前网格提取方法生成的网格，其质量显著劣于艺术家创作网格（Artist-Created Meshes，AMs，即人类艺术家制作的网格）。具体而言，当前网格提取方法依赖高密度面片，且忽略几何特征，由此引发效率低下、后处理流程繁琐以及表征质量欠佳等问题。为解决上述问题，我们提出MeshAnything模型：该模型将网格提取视作生成任务，可生成与指定形状对齐的艺术家创作网格。通过将任意三维表示形式下的三维资产转换为艺术家创作网格，MeshAnything可与各类三维资产制作方法集成，进而拓展其在三维产业中的应用场景。MeshAnything的架构由向量量化变分自编码器（VQ-VAE）与形状条件仅解码器Transformer构成。我们首先通过向量量化变分自编码器（VQ-VAE）学习网格词表，随后基于该词表训练形状条件仅解码器Transformer，实现形状条件自回归网格生成。我们开展的大量实验表明，本方法生成的艺术家创作网格面片数量仅为现有方法的数百分之一，在存储、渲染与仿真效率上实现大幅提升，同时精度可与既往方法媲美。