SensoryArt

The automatic recognition of sensory gestures in artworks provides the opportunity to open up methods of computational humanities to modern paradigms like sensory studies or everyday history. We introduce SensoryArt, a dataset of multisensory gestures in historical artworks, annotated with person boxes, pose estimation key points and gesture labels. We analyze algorithms for each label type and explore their combination for gesture recognition without intermediate supervision. These combined algorithms are evaluated for their ability to recognize and localize depicted persons performing sensory gestures. Our experiments show that direct detection of smell gestures is the most effective method for both detecting and localizing gestures. After applying post-processing, this method outperforms even image-level classification algorithms in image-level classification metrics, despite not being the primary training objective. This work aims to open up the field of sensory history to the computational humanities and provide humanities-based scholars with a solid foundation to complement their methodological toolbox with quantitative methods. Instruction on how to use the dataset for training and evaluating recognition algorithms can be found on GitHub: https://github.com/mathiaszinnen/SensoryGestureRecognition对艺术品中感官动作的自动识别，为计算人文科学（computational humanities）引入感官研究、日常史等现代研究范式提供了契机。 我们构建了SensoryArt数据集——一个面向历史艺术品多感官动作的标注数据集，其标注内容涵盖人物边界框、姿态估计关键点与动作标签。 我们针对各类标签类型设计了对应算法，并探索了无需中间监督的多算法融合方案以实现动作识别。针对融合算法的性能，我们开展了评估实验，检验其识别并定位艺术品中做出感官动作的人物的能力。实验结果表明，直接检测嗅觉动作的方案在动作检测与定位任务中均表现最优。在应用后处理流程后，尽管该方案并非以图像级分类为核心训练目标，但其在图像级分类指标上的表现甚至优于图像级分类算法。 本研究旨在为计算人文科学打开感官史研究领域的大门，并为人文学科研究者提供坚实的研究基础，使其能够将量化方法补充至自身的研究方法工具箱中。 关于使用该数据集训练与评估识别算法的操作指南，可访问以下GitHub仓库：https://github.com/mathiaszinnen/SensoryGestureRecognition