SHREC'14 Track: Large Scale Comprehensive 3D Shape Retrieval

Objective: The objective of this track is to evaluate the performance of 3D shape retrieval approaches on a large-sale comprehensive 3D shape database which contains different types of models, such as generic, articulated, CAD and architecture models. Introduction: With the increasing number of 3D models created every day and stored in databases, the development of effective and scalable 3D search algorithms has become an important research area. In this contest, the task will be retrieving 3D models similar to a complete 3D model query from a new integrated large-scale comprehensive 3D shape benchmark including various types of models. Owing to the integration of the most important existing benchmarks to date, the newly created benchmark is the most exhaustive to date in terms of the number of semantic query categories covered, as well as the variations of model types. The shape retrieval contest will allow researchers to evaluate results of different 3D shape retrieval approaches when applied on a large scale comprehensive 3D database. The benchmark is motivated by a latest large collection of human sketches built by Eitz et al. [1]. To explore how human draw sketches and human sketch recognition, they collected 20,000 human-drawn sketches, categorized into 250 classes, each with 80 sketches. This sketch dataset is exhaustive in terms of the number of object categories. Thus, we believe that a 3D model retrieval benchmark based on their object categorizations will be more comprehensive and appropriate than currently available 3D retrieval benchmarks to more objectively and accurately evaluate the real practical performance of a comprehensive 3D model retrieval algorithm if implemented and used in the real world. Considering this, we build a SHREC'14 Large Scale Comprehensive Track Benchmark (SHREC14LSGTB) by collecting relevant models in the major previously proposed 3D object retrieval benchmarks. Our target is to find models for as many as classes of the 250 classes and find as many as models for each class. These previous benchmarks have been compiled with different goals in mind and to date, not been considered in their sum. Our work is the first to integrate them to form a new, larger benchmark corpus for comprehensive 3D shape retrieval. Dataset: SHREC'14 Large Scale Comprehensive Retrieval Track Benchmark has 8,987 models, categorized into 171 classes. We adopt a voting scheme to classify models. For each classification, we have at least two votes. If these two votes agree each other, we confirm that the classification is correct, otherwise, we perform a third vote to finalize the classification. All the models are categorized according to the classifications in Eitz et al. [1], based on visual similarity. Evaluation Method: To have a comprehensive evaluation of the retrieval algorithm, we employ seven commonly adopted performance metrics in 3D model retrieval technique. Please cite the papers: [1] Bo Li, Yijuan Lu, Chunyuan Li, Afzal Godil, Tobias Schreck, Masaki Aono, Martin Burtscher, Qiang Chen, Nihad Karim Chowdhury, Bin Fang, Hongbo Fu, Takahiko Furuya, Haisheng Li, Jianzhuang Liu, Henry Johan, Ryuichi Kosaka, Hitoshi Koyanagi, Ryutarou Ohbuchi, Atsushi Tatsuma, Yajuan Wan, Chaoli Zhang, Changqing Zou. A Comparison of 3D Shape Retrieval Methods Based on a Large-scale Benchmark Supporting Multimodal Queries. Computer Vision and Image Understanding, November 4, 2014. [2] Bo Li, Yijuan Lu, Chunyuan Li, Afzal Godil, Tobias Schreck, Masaki Aono, Qiang Chen, Nihad Karim Chowdhury, Bin Fang, Takahiko Furuya, Henry Johan, Ryuichi Kosaka, Hitoshi Koyanagi, Ryutarou Ohbuchi, Atsushi Tatsuma. SHREC' 14 Track: Large Scale Comprehensive 3D Shape Retrieval. Eurographics Workshop on 3D Object Retrieval 2014 (3DOR 2014): 131-140, 2014.

目标：本赛道的目标是在包含通用型、铰接型、计算机辅助设计（Computer Aided Design, CAD）与建筑类模型等多种类型模型的大规模综合三维形状数据库上，评估三维形状检索方法的性能。
简介：随着每日生成并存储于数据库中的三维模型数量持续增长，开发高效且可扩展的三维搜索算法已成为重要的研究方向。本次竞赛的任务为：从包含多种模型类型的全新集成式大规模综合三维形状基准测试集中，检索与给定完整三维模型查询样本相似的三维模型。由于整合了截至目前所有主流的现有基准测试集，新构建的基准测试集在覆盖的语义查询类别数量与模型类型多样性两方面，均为当前最全面的数据集。本次三维形状检索竞赛可帮助研究者评估不同三维形状检索方法在大规模综合三维数据库上的应用效果。
该基准测试集的构建灵感来源于Eitz等人[1]发布的最新大型手绘草图数据集。为探究人类手绘草图的规律与草图识别方法，他们收集了20000幅手绘草图，划分为250个类别，每个类别包含80幅草图。该草图数据集在目标类别覆盖度上已达到极致全面。因此我们认为，基于其对象分类体系构建的三维模型检索基准测试集，将比现有各类三维检索基准测试集更为全面与适配，能够更客观、准确地评估三维模型检索算法在实际落地应用中的真实性能。
基于此，我们通过整合此前主流三维对象检索基准测试集中的相关模型，构建了SHREC'14大规模综合赛道基准测试集（SHREC14LSGTB）。我们的目标是尽可能覆盖全部250个类别，并为每个类别收集尽可能多的模型。此前的各类基准测试集均基于不同的设计目标构建，且截至目前尚未被整合为统一的数据集。本研究首次将这些基准测试集整合，形成了用于全面三维形状检索的全新大型基准测试语料库。
数据集：SHREC'14大规模综合检索赛道基准测试集共包含8987个模型，划分为171个类别。我们采用投票机制对模型进行分类：每个分类至少需要两轮投票，若两轮投票结果一致，则确认该分类正确；若结果不一致，则引入第三轮投票以最终确定分类结果。所有模型均基于视觉相似度，按照Eitz等人[1]提出的分类体系进行标注。
评价方法：为对检索算法开展全面评价，我们采用了三维模型检索领域中7种常用的性能指标。请引用以下文献：
[1] Bo Li, Yijuan Lu, Chunyuan Li, Afzal Godil, Tobias Schreck, Masaki Aono, Martin Burtscher, Qiang Chen, Nihad Karim Chowdhury, Bin Fang, Hongbo Fu, Takahiko Furuya, Haisheng Li, Jianzhuang Liu, Henry Johan, Ryuichi Kosaka, Hitoshi Koyanagi, Ryutarou Ohbuchi, Atsushi Tatsuma, Yajuan Wan, Chaoli Zhang, Changqing Zou. 《基于支持多模态查询的大规模基准测试集的三维形状检索方法对比》，《计算机视觉与图像理解》，2014年11月4日。
[2] Bo Li, Yijuan Lu, Chunyuan Li, Afzal Godil, Tobias Schreck, Masaki Aono, Qiang Chen, Nihad Karim Chowdhury, Bin Fang, Takahiko Furuya, Henry Johan, Ryuichi Kosaka, Hitoshi Koyanagi, Ryutarou Ohbuchi, Atsushi Tatsuma. 《SHREC'14赛道：大规模综合三维形状检索》，2014年欧洲图形学会三维对象检索研讨会（3DOR 2014），第131-140页，2014年。