EgoRoutine: Topic modelling for routine discovery from egocentric photo-streams

Developing tools to understand and visualize lifestyle is of high interest when addressing the improvement of habits and well-being of people. Routine, defined as the usual things that a person does daily, helps describe the individuals’ lifestyle. With this paper, we are the first ones to address the development of novel tools for automatic discovery of routine days of an individual from his/her egocentric images. In the proposed model, sequences of images are firstly characterized by semantic labels detected by pre-trained CNNs. Then, these features are organized in temporal-semantic documents to later be embedded into a topic models space. Finally, Dynamic-Time-Warping and Spectral-Clustering methods are used for final day routine/non-routine discrimination. Moreover, we introduce a new <em>EgoRoutine</em>-dataset, a collection of 104 egocentric days with more than 100.000 images recorded by 7 users. Results show that routine can be discovered and behavioural patterns can be observed.<br>Published in:Estefania Talavera, Carolin Wuerich, Nicolai Petkov, Petia Radeva, Topic modelling for routine discovery from egocentric photo-streams, Pattern Recognition, Volume 104, 2020, 107330, ISSN 0031-3203, https://doi.org/10.1016/j.patcog.2020.107330.

在致力于改善人类习惯与健康福祉的研究中，开发用于理解与可视化生活方式的工具具有重要研究价值。日常惯例（routine）被定义为个体每日的常规行为活动，可用于刻画个体的生活方式。在本文中，我们首次开展了基于个体第一人称视角图像（egocentric images）自动挖掘个体日常惯例时段的新型工具的开发研究。
在所提出的模型中，图像序列首先通过预训练卷积神经网络（Convolutional Neural Networks, CNNs）检测得到的语义标签进行特征表征。随后，将这些特征整理为时序-语义文档，进而嵌入至主题模型空间中。最终，采用动态时间规整（Dynamic Time Warping）与谱聚类（Spectral Clustering）方法完成日常/非日常惯例时段的最终判别。此外，我们还构建了全新的EgoRoutine数据集：该数据集包含7名用户录制的共计104个日常时段的第一人称视角图像，总图像数量超过10万张。实验结果表明，该方法可有效挖掘个体的日常惯例行为，并能观测到其行为模式。
本研究成果发表于：Estefania Talavera、Carolin Wuerich、Nicolai Petkov、Petia Radeva，《基于主题模型的第一人称照片流日常惯例挖掘》，《模式识别》，第104卷，2020年，文章编号107330，ISSN 0031-3203，https://doi.org/10.1016/j.patcog.2020.107330。