A Dataset of Thermal images of User Interfaces

Recent advancement in sensor technology facilitates having a thermal camera at a lower price. These cameras have many potential applications but can also be used for malicious purposes, such as capturing user interfaces and retrieving user information from heat traces in the thermal images. This dataset is created during an interactive study investigating the threat of thermal attacks on user interfaces. We adapted the following experimental setup during data collection. 2 camera perspectives- FLIR E8-XT camera placed behind the participant and Optris PI 450i camera placed left of the participant. 4 types of input devices- i) smartphone, ii) 3 keyboards- a PBT keyboard, an ABS keyboard, and a metal frame keyboard 3 types of user input data (text, email address, password) In summary, we have collected 1152 images from the FLIR camera and another 1152 images from the Optris camera through an interactive study with 32 participants. For each participant, we captured 36 images (9 types of user input, 4 types of input devices). The created dataset can be used to evaluate the deep learning model developed to prevent thermal imaging attacks. Furthermore, the ground truth user input of text, email address, and passwords are structured along with the corresponding image ID so that the advanced data-driven model can be employed to identify user input and investigate the type of user input that can be easily cracked using machine learning techniques.近年来传感器技术的进步使得热成像相机的购置成本显著降低。这类相机具备诸多潜在应用场景，但也可能被用于恶意用途，例如通过热成像图像中的热迹捕获用户界面并窃取用户信息。 本数据集源自一项针对用户界面热成像攻击威胁的交互式研究。数据采集阶段采用了如下实验设置：两台不同视角的热成像相机——FLIR E8-XT相机置于受试者身后，Optris PI 450i相机置于受试者左侧；四类输入设备，分别为智能手机，以及三款键盘：聚对苯二甲酸丁二醇酯（Polybutylene Terephthalate，PBT）键盘、丙烯腈-丁二烯-苯乙烯共聚物（Acrylonitrile Butadiene Styrene，ABS）键盘与金属框架键盘；三种用户输入数据类型：文本、电子邮箱地址与密码。 综上，本研究共招募32名受试者开展交互式实验，从FLIR相机采集得到1152张热成像图像，从Optris相机采集得到另外1152张图像。每名受试者对应36张图像（覆盖9种用户输入类型与4种输入设备）。 本数据集可用于评估针对热成像攻击的深度学习防御模型。此外，数据集已将文本、电子邮箱地址及密码的真实用户输入与对应图像ID进行关联标注，可支持先进数据驱动模型开展用户输入识别任务，并可用于探究哪些类型的用户输入更容易通过机器学习技术被破解。