Wrist-mounted IMU data towards the investigation of in-meal human eating behavior - the Food Intake Cycle (FIC) dataset

Introduction

The Food Intake Cycle (FIC) dataset was created by the Multimedia Understanding Group towards the investigation of in-meal eating behavior. The FIC dataset contains the triaxial acceleration and orientation velocity signals (\(6\) DoF) from \(21\) meal sessions provided by \(12\) unique subjects. All meals were recorded in the restaurant of Aristotle University of Thessaloniki using a commercial smartwatch, the Microsoft Band \(2\)™ for ten out of the twenty-one meals and the Sony Smartwatch \(2\)™ for the remaining meals. In addition, the start and end moments of each food intake cycle as well as of each micromovement are annotated throughout the FIC dataset.

Description

A total of \(12\) subjects were recorded while eating their launch at the university’s cafeteria. The total duration of the \(21\) meals sums up to \(246\) minutes, with a mean duration of \(11.7\) minutes. Each participant was free to select the food of their preference, typically consisting of a starter soup, a salad, a main course and a desert. Prior to the recording, the participant was asked to wear the smartwatch to the hand that he typically uses in his everyday life to manipulate the fork and/or the spoon. A GoPro™ Hero \(5\) camera was already set at the table of the participant using a small, \(23\) cm in height, tripod facing the participant, including both the food tray and upper body part in it’s field of view. The purpose of video recording was to obtain ground truth data by manually annotating the IMU sequences based on the video stream. Participants were also asked to perform a clapping hand movement both at the start and end of the meal, for synchronization purposes (as this movement is distinctive in the accelerometer signal). No other instructions were given to the participants. It should be noted that the FIC dataset does not contain instances related with liquid consumption or eating without the fork, knife and spoon (e.g. eating directly with hands). The accompanying python script viz_dataset.py will visualize the IMU signals and food intake cycle (i.e., bite) ground truth intervals for each of the recordings. Information on how to execute the Python scripts can be found below.

# The script(s) and the pickle file must be located in the same directory.
# Tested with Python 3.6.4
# Requirements: Numpy, Pickle and Matplotlib

# Visualize signals and ground truth
$ python viz_dataset.py

FIC is also tightly related to FreeFIC, a dataset we created in order to investigate the in-the-wild eating behavior. More information on FreeFIC can be found here and here.

Annotation

Micromovements

For all recordings, the start and end points of all \(6\) micromovements of interest were manually labeled. The micromovements of interest include:

pick food, wrist manipulates a fork to pick food from the plate

upwards, wrist moves upwards, towards the mouth area

downwards, wrist moves downwards, away from the mouth area

mouth, wrist inserts food in mouth

no movement, wrist exhibits no movement

other movement, every other wrist movement

The annotation process was performed in such a way that the start and end times of each micro-movement span the whole meal session, without overlapping each other.

Food intake cycles

For all recordings, we annotated the start and end points for each intake cycle (i.e. every bite). Each food intake cycle starts with a p, ends with a d and contains an m micromovement.

Publications

If you plan to use the FIC dataset or any of the resources found in this page, please cite our work:

@article{kyritsis2019modeling,
title={Modeling Wrist Micromovements to Measure In-Meal Eating Behavior from Inertial Sensor Data},
author={Kyritsis, Konstantinos and Diou, Christos and Delopoulos, Anastasios},
journal={IEEE journal of biomedical and health informatics},
year={2019},
publisher={IEEE}}

@inproceedings{kyritsis2017food,
title={Food intake detection from inertial sensors using lstm networks},
author={Kyritsis, Konstantinos and Diou, Christos and Delopoulos, Anastasios},
booktitle={International Conference on Image Analysis and Processing},
pages={411--418},
year={2017},
organization={Springer}}

@inproceedings{kyritsis2017automated,
title={Automated analysis of in meal eating behavior using a commercial wristband IMU sensor},
author={Kyritsis, Konstantinos and Tatli, Christina Lefkothea and Diou, Christos and Delopoulos, Anastasios},
booktitle={2017 39th Annual International Conference of the IEEE Engineering in Medicine and Biology Society (EMBC)},
pages={2843--2846},
year={2017},
organization={IEEE}}

Technical details

We provide the FIC dataset as a pickle. The file can be loaded using Python in the following way:

import pickle as pkl
import numpy as np

with open('./FIC.pkl','rb') as fh:
dataset = pkl.load(fh)

The dataset variable in the snipet above is a dictionary with \(6\) keys. Namely:

'subject_id'

'session_id'

'signals_raw'

'signals_proc'

'meal_gt'

'bite_gt'

The contents under a specific key can be obtained by:

sub = dataset['subject_id'] # for the subject id
ses = dataset['session_id'] # for the session id
raw = dataset['signals_raw'] # for the raw IMU signals
proc = dataset['signals_proc'] # for the processed IMU signals
mm = dataset['mm_gt'] # for the micromovement ground truth
bite = dataset['bite_gt'] # for the bite ground truth

The sub, ses, raw, proc, mm and gt variables in the snipet above are lists with a length equal to \(21\). Elements across all lists are aligned; e.g., the 3rd element of the list under the 'session_id' key corresponds to the 3rd element of the list under the 'signals_proc' key.

sub: list
Each element of the sub list is a scalar (integer) that corresponds to the unique identifier of the subject that can take values between \(1\) and \(12\). Moreover, the subject identifier in FIC is in-line with the subject identifier in the FreeFIC dataset (information available here and here); i.e., FIC’s subject with id equal to 2  is the same person as FreeFIC’s subject with id equal to 2.

ses: list
Each element of this list is a scalar (integer) that corresponds to the unique identifier of the session that can range between 1 and \(3\). It should be noted that not all subjects have the same number of sessions.

raw: list
Each element of this list is dictionary with the 'acc', 'gyr' and 'offset' keys.
The data under the 'acc' key is a \(N_{acc}\times4\) numpy.ndarray that contains the timestamps in seconds (first column) and the \(3D\) raw accelerometer measurements in \(g\) (second, third and forth columns - representing the \(x, y\) and \(z\) axis, respectively). The data under the 'gyr' key is a \(N_{gyr} \times 4\) numpy.ndarray that contains the timestamps in seconds (first column) and the \(3D\) raw gyroscope measurements in \(degrees/second\)(second, third and forth columns - representing the \(x, y\) and \(z\) axis, respectively). All sensor streams are transformed in such a way that reflects all participants wearing the smartwatch at the same hand with the same orientation, thusly achieving data uniformity. This transformation is in par with the signals in the FreeFIC dataset (information available here and here). Finally, the length of the raw accelerometer and gyroscope numpy.ndarrays is different \(N_{acc} \neq N_{gyr}\). This behavior is predictable and is caused by the Android/MS Band platforms. The offset key contains a float that is used to align the IMU sensor streams with the videos that were used for annotation purposes (videos are not provided).

proc: list
Each element of this list is an \(M \times 7\)  numpy.ndarray that contains the timestamps, \(3D\) accelerometer and gyroscope measurements for each meal. Specifically, the first column contains the timestamps in seconds, the second, third and forth columns contain the \(x,y\) and \(z\) accelerometer values in \(g\) and the fifth, sixth and seventh columns contain the \(x, y\) and \(z\) gyroscope values in \(degrees/second\). Unlike elements in the raw list, processed measurements (in the proc list) have a constant sampling rate of 100 Hz and the accelerometer/gyroscope measurements are aligned with each other. In addition, all sensor streams are transformed in such a way that reflects all participants wearing the smartwatch at the same hand with the same orientation, thusly achieving data uniformity. This transformation is in par with the signals in the FreeFIC dataset (information available here and here). No other preprocessing is performed on the data; e.g., the acceleration component due to the Earth's gravitational field is present at the processed acceleration measurements. The potential researcher can consult the article "Modeling Wrist Micromovements to Measure In-Meal Eating Behavior from Inertial Sensor Data" by Kyritsis et al. on how to further preprocess the IMU signals (i.e., smooth and remove the gravitational component).

mm: list
Each element of this list is a \(K \times 3\) numpy.ndarray. Each row represents a single micromovement interval. The first column contains the timestamps of the start moments in seconds, the second column the timestamps of the end moments in seconds and the third column a number representing the type of the micromovement. The identifier to micromovement mapping is provided below:
\([1] \rightarrow\) no movement
\([2] \rightarrow\) upwards
\([3] \rightarrow\) downwards
\([4] \rightarrow\) pick food
\([5] \rightarrow\) mouth
\([6] \rightarrow\) other movement

bite: list
Each element of this list is a \(L\times2\) numpy.ndarray. Each row represents a single food intake event (i.e., a bite). The first column contains the start moments while the second column contains the end moments of each intake event. Both the start and end moments are provided in seconds.

Ethics and funding

Informed consent, including permission for third-party access to anonymised data, was obtained from all subjects prior to their engagement in the study. The work has received funding from the European Union's Horizon 2020 research and innovation programme under Grant Agreement No 727688 - BigO: Big data against childhood obesity.

Contact

Any inquiries regarding the FIC dataset should be addressed to:

Dr. Konstantinos KYRITSIS

Multimedia Understanding Group (MUG)
Department of Electrical & Computer Engineering
Aristotle University of Thessaloniki
University Campus, Building C, 3rd floor
Thessaloniki, Greece, GR54124

Tel: [phone], 996365 
Fax: [phone]
E-mail: kokirits [at] mug [dot] ee [dot] auth [dot] gr

# 引言
进食周期（Food Intake Cycle, FIC）数据集由多媒体理解组（Multimedia Understanding Group）构建，旨在探究进餐过程中的进食行为。该数据集包含来自12名不同受试者的21次进餐记录的三轴加速度与定向速度信号（6自由度，6 DoF）。所有进餐记录均在塞萨洛尼基亚里士多德大学餐厅完成，其中21次进餐中有10次使用商用智能手表微软手环2™（Microsoft Band 2™）采集数据，剩余11次使用索尼智能手表2™（Sony Smartwatch 2™）采集数据。此外，FIC数据集已对所有进食周期与每一次微动作的起止时刻进行人工标注。

# 数据集详情
共有12名受试者在大学食堂进食午餐时被采集数据。21次进餐的总时长为246分钟，平均每次进餐时长为11.7分钟。每位受试者可自由选择喜爱的餐食，通常包括开胃汤、沙拉、主菜与甜点。采集数据前，受试者需将智能手表佩戴于日常使用叉子和/或勺子的惯用手。实验已提前在受试者餐桌旁架设一台GoPro™ Hero 5相机，搭配高度为23cm的小型三脚架，相机朝向受试者，拍摄范围涵盖餐盘与受试者上半身。视频录制的目的是通过基于视频流人工标注惯性测量单元（Inertial Measurement Unit, IMU）序列，获取真实标注（Ground Truth）数据。此外，受试者需在进餐开始与结束时各完成一次拍手动作，用于时间同步（该动作在加速度计信号中特征明显）。未向受试者提供其他额外实验要求。需注意，FIC数据集未包含液体摄入或不使用刀叉勺进食（如直接用手进食）的相关样本。

随附的Python脚本viz_dataset.py可可视化各次记录的IMU信号与进食周期（即单次咬食）的真实标注区间。Python脚本的运行说明详见下文：
# 脚本与pickle文件需置于同一目录下。
# 已在Python 3.6.4环境下测试通过。
# 依赖库：Numpy、Pickle与Matplotlib
# 可视化信号与真实标注
$ python viz_dataset.py

FIC数据集还与我们为探究野外进食行为构建的FreeFIC数据集密切相关。FreeFIC数据集的详细信息可参见此处与此处。

# 标注说明
## 微动作类别
针对所有记录，6类目标微动作的起止点均已完成人工标注。目标微动作类别如下：
- 取食：手腕操控叉子从餐盘夹取食物
- 向上移动：手腕向上移动至口腔区域
- 向下移动：手腕向下远离口腔区域
- 入口：手腕将食物送入口中
- 无动作：手腕无任何运动
- 其他动作：其余所有手腕动作

标注过程确保每一类微动作的起止时间覆盖整个进餐时段，且各微动作区间互不重叠。

## 进食周期标注
针对所有记录，我们已标注每一次进食周期（即单次咬食）的起止时刻。每个进食周期以p类动作起始，以d类动作收尾，并包含m类微动作区间。

# 引用要求
若您计划使用FIC数据集或本页面提供的任何资源，请引用以下论文：
@article{kyritsis2019modeling,
title={Modeling Wrist Micromovements to Measure In-Meal Eating Behavior from Inertial Sensor Data},
author={Kyritsis, Konstantinos and Diou, Christos and Delopoulos, Anastasios},
journal={IEEE journal of biomedical and health informatics},
year={2019},
publisher={IEEE}}

@inproceedings{kyritsis2017food,
title={Food intake detection from inertial sensors using lstm networks},
author={Kyritsis, Konstantinos and Diou, Christos and Delopoulos, Anastasios},
booktitle={International Conference on Image Analysis and Processing},
pages={411--418},
year={2017},
organization={Springer}}

@inproceedings{kyritsis2017automated,
title={Automated analysis of in meal eating behavior using a commercial wristband IMU sensor},
author={Kyritsis, Konstantinos and Tatli, Christina Lefkothea and Diou, Christos and Delopoulos, Anastasios},
booktitle={2017 39th Annual International Conference of the IEEE Engineering in Medicine and Biology Society (EMBC)},
pages={2843--2846},
year={2017},
organization={IEEE}}

# 技术细节
我们以pickle文件格式提供FIC数据集，可通过以下Python代码加载：
import pickle as pkl
import numpy as np

with open('./FIC.pkl','rb') as fh:
dataset = pkl.load(fh)

上述代码中的dataset变量为包含6个键的字典，具体键名如下：
- 'subject_id'
- 'session_id'
- 'signals_raw'
- 'signals_proc'
- 'mm_gt'
- 'bite_gt'

可通过以下方式获取指定键对应的内容：
sub = dataset["subject_id"] # 获取受试者ID
ses = dataset["session_id"] # 获取会话ID
raw = dataset["signals_raw"] # 获取原始IMU信号
proc = dataset["signals_proc"] # 获取预处理后的IMU信号
mm = dataset["mm_gt"] # 获取微动作真实标注
bite = dataset["bite_gt"] # 获取咬食真实标注

上述代码中的sub、ses、raw、proc、mm与bite变量均为长度为21的列表，各列表元素严格对齐；例如，'session_id'键对应列表的第3个元素与'signals_proc'键对应列表的第3个元素属于同一次进餐记录。

### sub列表
列表中每个元素为1至12之间的整数标量，对应受试者的唯一标识符。此外，FIC数据集的受试者ID与FreeFIC数据集的受试者ID保持一致（详细信息参见此处与此处）；即，FIC数据集中ID为2的受试者与FreeFIC数据集中ID为2的受试者为同一人。

### ses列表
列表中每个元素为1至3之间的整数标量，对应进餐会话的唯一标识符。需注意，并非所有受试者的进餐会话数量均相同。

### raw列表
列表中每个元素为包含'acc'、'gyr'与'offset'三个键的字典。
'acc'键对应的数据为形状为$N_{acc} imes 4$的numpy数组，第一列为以秒为单位的时间戳，第二至第四列为3轴原始加速度计测量值（单位为g，分别对应x、y、z轴）。
'gyr'键对应的数据为形状为$N_{gyr} imes 4$的numpy数组，第一列为以秒为单位的时间戳，第二至第四列为3轴原始陀螺仪测量值（单位为度/秒，分别对应x、y、z轴）。
所有传感器数据流均经过统一转换，确保所有受试者的智能手表佩戴手与朝向完全一致，以实现数据标准化。该转换方式与FreeFIC数据集的信号处理方式一致（详细信息参见此处与此处）。需注意，原始加速度计与陀螺仪数据的数组长度并不相同（$N_{acc}
eq N_{gyr}$），该现象符合预期，由Android系统与微软手环平台的差异导致。'offset'键对应的值为浮点型标量，用于将IMU传感器数据流与标注所用的视频进行时间对齐（视频数据未随数据集一并提供）。

### proc列表
列表中每个元素为形状为$M imes7$的numpy数组，包含单次进餐的时间戳、3轴加速度计与陀螺仪测量值。具体来说，第一列为以秒为单位的时间戳，第二至第四列为x、y、z轴加速度计测量值（单位为g），第五至第七列为x、y、z轴陀螺仪测量值（单位为度/秒）。
与raw列表中的原始数据不同，proc列表中的预处理数据采样率固定为100 Hz，且加速度计与陀螺仪数据已完成时间对齐。此外，所有传感器数据流均经过统一转换，确保所有受试者的智能手表佩戴手与朝向完全一致，以实现数据标准化。该转换方式与FreeFIC数据集的信号处理方式一致（详细信息参见此处与此处）。该数据集未进行其他预处理操作；例如，预处理后的加速度数据仍包含地球重力场产生的加速度分量。有需要的研究者可参考Kyritsis等人发表的论文《Modeling Wrist Micromovements to Measure In-Meal Eating Behavior from Inertial Sensor Data》，了解如何对IMU信号进行进一步预处理（如平滑处理与去除重力分量）。

### mm列表
列表中每个元素为形状为$K imes3$的numpy数组，每一行代表一个微动作区间。第一列为微动作起始时刻的时间戳（单位为秒），第二列为结束时刻的时间戳，第三列为微动作类型的编码值。微动作类型与编码的对应关系如下：
[1] → 无动作
[2] → 向上移动
[3] → 向下移动
[4] → 取食
[5] → 入口
[6] → 其他动作

### bite列表
列表中每个元素为形状为$L imes2$的numpy数组，每一行代表一次进食事件（即单次咬食）。第一列为进食事件的起始时刻，第二列为结束时刻，时间单位均为秒。

# 伦理与资助
所有受试者在参与研究前均已签署知情同意书，其中包含允许第三方访问匿名化数据的条款。本研究获得欧盟地平线2020研究与创新计划资助，项目编号为727688——BigO：对抗儿童肥胖的大数据项目。

# 联系方式
若您对FIC数据集有任何疑问，请联系：
康斯坦丁诺斯·基里齐斯博士（Dr. Konstantinos KYRITSIS）
多媒体理解组（Multimedia Understanding Group, MUG）
电气与计算机工程系
塞萨洛尼基亚里士多德大学
大学校园C楼3层
塞萨洛尼基，希腊，GR54124
电话：[phone], 996365
传真：[phone]
邮箱：kokirits [at] mug [dot] ee [dot] auth [dot] gr