A Dataset of Thermal images of User Interfaces

Recent advancement in sensor technology facilitates having a thermal camera at a lower price. These cameras have many potential applications but can also be used for malicious purposes, such as capturing user interfaces and retrieving user information from heat traces in the thermal images. This dataset is created during an interactive study investigating the threat of thermal attacks on user interfaces. We adapted the following experimental setup during data collection.

2 camera perspectives- FLIR E8-XT camera placed behind the participant and Optris PI 450i camera placed left of the participant. 

4 types of input devices- i) smartphone, ii) 3 keyboards- a PBT keyboard, an ABS keyboard, and a metal frame keyboard

3 types of user input data (text, email address, password)

In summary, we have collected 1152 images from the FLIR camera and another 1152 images from the Optris camera through an interactive study with 32 participants. For each participant, we captured 36 images (9 types of user input, 4 types of input devices). The created dataset can be used to evaluate the deep learning model developed to prevent thermal imaging attacks. Furthermore, the ground truth user input of text, email address, and passwords are structured along with the corresponding image ID so that the advanced data-driven model can be employed to identify user input and investigate the type of user input that can be easily cracked using machine learning techniques.

传感器技术的最新进展使得热成像相机（thermal camera）的购置成本显著降低。这类相机虽具备诸多潜在应用场景，但也可能被用于恶意用途，例如通过热成像画面中的热量痕迹捕获用户界面并窃取用户信息。本数据集源自一项针对用户界面热成像攻击威胁的交互式研究，数据采集过程采用了如下实验设置：
- 两种拍摄视角：将FLIR E8-XT热成像相机置于受试者身后，Optris PI 450i热成像相机置于受试者左侧；
- 四类输入设备：①智能手机；②三种键盘——分别为PBT材质键盘、ABS材质键盘与金属框架键盘；
- 三类用户输入数据：文本、电子邮箱地址与密码。

综上，本研究共招募32名受试者开展交互式实验，从FLIR相机与Optris相机中分别采集到1152张热成像图像。每名受试者对应36张图像（涵盖9种用户输入类型与4类输入设备）。本数据集可用于评估针对热成像攻击的防御深度学习模型。此外，数据集已将文本、电子邮箱地址与密码的真实标注（ground truth）与对应图像ID进行结构化关联，以便先进的数据驱动模型可用于识别用户输入，并探究哪些类型的用户输入更容易通过机器学习技术被破解。