EgoFoodPlaces: Hierarchical Approach to Classify Food Scenes in Egocentric Photo-Streams

Recent studies have shown that the environment where people eat can affect their nutritional behaviour. In this work, we provide automatic tools for personalised analysis of a person's health habits by the examination of daily recorded egocentric photo-streams. Specifically, we propose a new automatic approach for the classification of food-related environments, that is able to classify up to 15 such scenes. In this way, people can monitor the context around their food intake in order to get an objective insight into their daily eating routine. We propose a model that classifies food-related scenes organized in a semantic hierarchy. Additionally, we present and make available a new egocentric dataset composed of more than 33000 images recorded by a wearable camera, over which our proposed model has been tested. Our approach obtains an accuracy and F-score of 56% and 65%, respectively, clearly outperforming the baseline methods.<br>Published in:E. T. Martinez, M. Leyva-Vallina, M. M. K. Sarker, D. Puig, N. Petkov and P. Radeva, "Hierarchical Approach to Classify Food Scenes in Egocentric Photo-Streams," in <em>IEEE Journal of Biomedical and Health Informatics</em>, vol. 24, no. 3, pp. 866-877, March 2020, doi: 10.1109/JBHI.2019.2922390.<br>

现有研究表明，人们进食时所处的环境会影响其营养摄入行为。本研究通过分析日常记录的第一人称视角照片流（egocentric photo-streams），为个体健康习惯的个性化分析提供了自动化工具。具体而言，我们提出了一种全新的自动化方法用于饮食相关环境分类，该方法可对多达15类此类场景进行分类。借此，人们可监测自身进食时的周边场景，从而客观洞悉自身的日常饮食规律。我们提出了一种基于语义层级结构的饮食相关场景分类模型。此外，我们构建并公开了一个全新的第一人称视角照片数据集，该数据集包含超过33000张由可穿戴相机拍摄的图像，我们已基于该数据集对所提模型开展了测试。我们的方法准确率达56%，F1值（F-score）达65%，性能显著优于基线方法。<br>发表于：E. T. 马丁内斯、M. 莱瓦-巴里亚、M. M. K. 萨克尔、D. 普伊格、N. 佩特科夫与P. 拉德瓦，《面向第一人称视角照片流的饮食场景分层分类方法》，载于《IEEE生物医学与健康信息学汇刊》，2020年3月，第24卷第3期，第866-877页，DOI：10.1109/JBHI.2019.2922390。