Benchmark Webcam Eye-Tracking Software

These data correspond to the industrial systematic review and benchmark test set related to the paper A Benchmark Study of Webcam Eye Tracking Software in Robotic Process Automation. The industrial systematic review is an approach adapted from Kitchenham's method that gathers all the eye tracking tools obtained after running a search in search engines, entering defined queries, and filtering by inclusion/exclusion criteria that we define in our planning. More precisely, this review includes all the different results obtained from the search queries (Eye tracking or Gaze tracking or Eye tracker) and (software or app or tool or github) entered in the search engines DuckDuckgo, Bing and Google. After the search, the eye tracking software results will be filtered according to the exclusion/inclusion criteria (C): · C1: The software must be available and documented or supported. · C2: The software must work on desktop computers or laptops, excluding mobile devices. · C3: The software should be open-source and/or free software, and if it is not, a demo version should be available even if it is for a trial period. · C4: The software must provide the option to use the built-in/native camera of a PC or webcam as hardware for eye tracking. It is excluded head-mounted devices, Virtual Reality glasses and commercial eye tracker hardware. · C5: The software must provide Eye tracking as software, not as a feature or extension of a service unrelated about specifically predicting the POG as functionality. · C6: The software must be installable in the established benchmark test setup. The software that passes the inclusion/exclusion criteria are those evaluated in a benchmark. For this evaluation, there are 4 different objects where eye tracking software tests are performed. These objects are called Circle, Buttons, Cartesian system and Email Invoice and differ from each other in the elements and location of the targets for testing each eye tracking software. The results obtained from tests objects are strategically differentiated into two distinct categories to becnhmark eye tracking software according to Accuracy, Precision and POG Coincidence metrics: · Accuracy and precision. Measures directly the accuracy and precision of the eye tracking software in the Cartesian system object. o Accuracy. It is quantified as the angular error between the point representing the centroid of the estimated POG from the eye tracking software and the point representing the centroid of the target, at which the subject is gazing. o Precision. It is calculated using the root mean square (RMS) of the sampled points obtained from the screenshots of one test session. · POG coincidence. Classifies where the estimated POG is in each screenshot. This category applies to Circle, Buttons and Invoice email objects. o Match on target (green circle with a diameter of 1cm). The estimated POG centroid by the eye tracking software in the screenshot is inside the green circle area or touches its circumference or limit. o Match in a close target (blue circle with a radius of 2cm that contains the green circle (target) inscribed within it. The estimated POG centroid by the eye tracking software in the screenshot is inside the blue circle area or touches its circumference or limit. o Off-target. The estimated POG centroid by the eye tracking software in the screenshot is outside the target limit The structure of the data packages is organized as follows: SLR_eye_tracking_software.xlsx: This .xlsx file corresponds entirely to the entire industrial systematic review on eye tracking software. It consists of 3 sheets: SLR: This encapsulates the eye tracking software acquired through search queries inputted across diverse search engines. It is composed of 4 columns. SOFTWARE: Web platform or application that provides the eye tracking software. SOFTWARE URL: URL to the repository or host page of the software. SEARCH URL: URL of the page mentioning the software QUERY SEARCH: Keywords used for the search and entered in the search engines. Conducting: This page shows the classification of eye tracking software according to the established inclusion/exclusion criteria, where each column corresponds to one of the criteria. Software that does not pass the successive criteria is discarded. Columns are as follows: AVAILABILITY; DOCUMENTATION AND SUPPORT (C1): Software that is downloadable and supported for viable use (Available, Not Available). Documentation and/or support about the software setup, run and outputs (Yes, No). PLATFORM (C2): Type of device and operating system where the sofware is run (PC: Windows, Ubuntu, MacOs, ...; Mobile: iOs, Android ...; Multiplatform). LICENSE (C3): OPENSOURCE: Public access to the source code (Yes, No). FREE SOFTWARE: It can be downloaded, used and distributed free of charge for non-commercial purposes (Yes, No). TRIAL DEMO: Availability of trial version if the software is neither open source nor free (Yes, No, -). *If it is opensource, trial demo value is “-“. HARDWARE (C4): Hardware required to run the software (Commercial, Webcam, Both, It is not determined). PURPOSE (C5): Eye tracking purposes or functionalities (Predict POG, AOI generation, Head motion capture, ...). BENCHMARK SETUP COMPATIBILITY (C6): The software shows compatibility with the benchmark setup (Yes, No). All Tools: On this page, all the information obtained from each of the eye tracking software is displayed, without discarding based on the inclusion/exclusion criteria. In addition to the columns described on the conducting page, additional columns with extra information are included. SOFTWARE END-USE CONTEXT: Purpose of use of the software (Generic, Specific). SOURCE CODE: Eye tracker source code language (Java, C++, Python, JavaScript or it is not determined). INSTALLATION: Installation of the software in the system (Executable, SaaS, WebApp, Compiled, ...). benchmark.xlsx: This file represents the results of the benchmark carried out on eye tracking software. It consists of two sheets. · Accuracy + Precision: It corresponds to the results obtained from the POG accuracy and precision tests conducted based on the Cartesian system object. Each column represents the following data: o Software: Eye tracking software tested. o Test ID: Test identification. o Screenshot ID: Screenshot identification. o Centroid POG X: X-coordinate of the centroid of the (POG) estimated by the eye tracking software. o Centroid POG Y: Y-coordinate of the centroid of the (POG) estimated by the eye tracking software. o Target X: X-coordinate of the centroid of the target. o Target Y: Y-coordinate of the centroid of the target. o Distance X (px): Distance from the Centroid POG X to Target X (in pixels). o Distance Y (px): Distance from the Centroid POG Y to Target Y (in pixels). o Euc.Distance (px): Euclidean distance (in pixels) from the centroid of the POG to the centroid of the target point. o Euc.Distance (cm): Euclidean distance (in centimeters) from the centroid of the POG to the centroid of the target point. o Average Euc.Distance (px): Average Euclidean distance (in pixels) from the centroid of the Point of Gaze (POG) to the centroid of the target point, referred to the same test. o Average euc.Distance (cm): Average Euclidean distance (in centimeters) from the centroid of the Point of Gaze (POG) to the centroid of the origin point, referred to the same test. o Precision: Value of the Root Mean Square (RMS) of the average distance from the centroid of the Point of Gaze (POG) to the centroid of the target point. o Accuracy (º): Accuracy estimated (in degrees) according to the setup (60 centimeters and 0º eyes to screen target distance). o Accuracy per points (º): Accuracy estimated (in degrees) per points referred to the same Test ID o Average Accuracy per points (º): Average Accuracy (in degrees) per points referred to the same Test ID. · Coincidence: It corresponds to the results of the benchmark from the POG coincidence tests conducted based on Circle, Buttons and Invoice Email objects. Each column represents the following data: o Software: Eye tracking software tested. o Scenario: Object tested. o Test ID: Test identification. o Match Target: Number of Match target POG. o Close Target: Number of close target POG o Off Target: Number of Off target POG. o Match Target + Close Target: Sum of the match target POG and Close Target POG. o Total: Number of total POG: Total number of recounted points (50). o %Match Target: Percentage of match target POG. o %Close Target: Percentage of close target POG. o %Off Target: Percentage of off target POG. o %Match Target + Close Target: Sum of the match target POG and close target POG percentage. benchmark_tests/ · tests_instructions.txt It is an instruction manual containing information on how the tests are conducted in each object. · tests_scenarios/ This folder contains the four objects designed for the benchmark evaluation. o circle.html o buttons.html o cartesian_system.html o invoice_email.html · eye-tracking_software_tests/ o GazeRecorder/ - buttons/ · T1/ o X_Y_Z.txt o Recording_XXXXXXXXX_YYYY.mht · T2/ o X_Y_Z.txt o Recording_XXXXXXXXX_YYYY.mht · T3/ o X_Y_Z.txt o Recording_XXXXXXXXX_YYYY.mht - circle/… - invoice_email/… - cartesian_system/ · T1/ o Recording_XXXXXXXXX_YYYY.mht o 1.png o 2.png o … o 20.png · T2/ o Recording_XXXXXXXXX_YYYY.mht o 1.png o 2.png o … o 20.png - (*) POG_GazeRecorder.png The folder eye tracking_software_tests contains the results of the tests in a stratified manner. At the first level, there are folders with data belonging to the respective eye tracking software to be tested (GazeRecorder, Eyedid, and Webgazer.js). At the next level, corresponding to each software, there would be folders for the test objects (buttons, circle, cartesian_system, and invoice_email). Within each object test folder, we find folders corresponding to the Test ID. Within each Test ID, there are two files: · X_Y_Z.txt. o X represents the number of match target POG estimated by the eye tracking software. o Y represents the number of of close target POG estimated by the eye tracking software. o Z represents the number of off-target POG estimated by the eye tracking software. · Recording_XXXXXXXXX_YYYY.mht. o It is the User Interface log generated by the stepRecorder application for Windows. It contains 30 screenshots about the POG estimation for the coincidence tests objects (Buttons, Circle and Invoice Email) and 20 screenshots for the accuracy and precision test object (Cartesian System). · 1.png to 20.png o These screenshots are exported from the .mht file in order to obtain the centroid of the Point of Gaze (POG) and calculate accuracy and precision accurately, as it is not possible directly from the .mht file. (*) The Point of Gaze (POG) in GazeRecorder is not displayed as a small dot, as is common and as it happens in EyeDid and Webgazer.js, but rather as an eye encapsulated in a rectangle. POG_GazeRecorder.png shows the centroid for the POG estimated by that eye tracking software.

本数据集对应于论文《A Benchmark Study of Webcam Eye Tracking Software in Robotic Process Automation》相关的工业系统综述与基准测试集。本次工业系统综述采用适配于Kitchenham方法的研究范式，通过搜索引擎检索、输入预设查询词，并按照研究规划中制定的纳入/排除标准进行筛选，从而收集所有符合条件的眼动追踪（Eye tracking）工具。

更具体而言，本次综述涵盖了在DuckDuckgo、Bing、Google搜索引擎中输入检索词"(Eye tracking or Gaze tracking or Eye tracker) and (software or app or tool or github)"得到的所有检索结果。检索完成后，将按照以下纳入/排除标准（C1-C6）对眼动追踪软件结果进行筛选：
· C1：软件必须可获取且具备文档或技术支持；
· C2：软件需适配台式电脑或笔记本电脑，排除移动设备；
· C3：软件应为开源软件和/或免费软件；若不符合，则需提供试用版（即便仅在试用期限内可用）；
· C4：软件必须支持使用个人电脑内置摄像头或网络摄像头作为眼动追踪的硬件，本标准排除头戴式设备、虚拟现实（Virtual Reality, VR）眼镜及商用眼动仪（Eye tracker）硬件；
· C5：软件必须将眼动追踪作为核心软件功能，而非作为与专门以预测注视点（Point of Gaze, POG）为核心功能无关的服务的附属特性或扩展；
· C6：软件必须可在预设的基准测试环境中安装。

通过上述筛选标准的软件将进入基准测试环节。本次测试共包含4类测试对象，分别为圆形靶标（Circle）、按钮靶标（Buttons）、笛卡尔坐标系靶标（Cartesian system）与邮件发票靶标（Email Invoice），各类靶标的测试元素与目标位置各不相同。

测试结果将根据准度（Accuracy）、精度（Precision）与注视点（POG）重合度三类指标，分为两大类别以完成眼动追踪软件的基准评估：
1. 准度与精度：直接测试眼动追踪软件在笛卡尔坐标系靶标中的准度与精度
- 准度：量化为眼动追踪软件估算的注视点（POG）质心与被试注视目标的质心之间的角度误差；
- 精度：通过单次测试会话的截图采样点的均方根（Root Mean Square, RMS）值计算得出。
2. 注视点（POG）重合度：对每张截图中估算的注视点（POG）位置进行分类，该分类适用于圆形靶标、按钮靶标及邮件发票靶标：
- 靶内匹配：估算的注视点（POG）质心位于绿色圆形区域内或与其圆周/边界相切（绿色圆形直径为1cm）；
- 近靶匹配：估算的注视点（POG）质心位于蓝色圆形区域内或与其圆周/边界相切（蓝色圆形半径为2cm，内部内嵌绿色靶标圆形）；
- 靶外匹配：估算的注视点（POG）质心位于靶标边界之外。

### 数据包结构
1. **SLR_eye_tracking_software.xlsx**：该Excel文件完整收录了所有关于眼动追踪软件的工业系统综述内容，包含3个工作表：
- **SLR**：收录通过多搜索引擎检索得到的眼动追踪软件信息，共4列：
- SOFTWARE：提供眼动追踪软件的网络平台或应用程序；
- SOFTWARE URL：该软件的代码仓库或宿主页面链接；
- SEARCH URL：提及该软件的检索结果页面链接；
- QUERY SEARCH：用于检索并输入至搜索引擎的关键词。
- **Conducting**：该页面展示了根据预设纳入/排除标准对眼动追踪软件进行的分类结果，每一列对应一项筛选标准，未通过连续筛选标准的软件将被剔除。各列说明如下：
- AVAILABILITY; DOCUMENTATION AND SUPPORT (C1)：软件是否可下载并可用于实际使用（分为"可用""不可用"）；是否具备软件安装、运行及输出结果相关的文档和/或技术支持（分为"是""否"）；
- PLATFORM (C2)：软件运行所需的设备及操作系统类型（PC：Windows、Ubuntu、MacOS等；移动设备：iOS、Android等；多平台）；
- LICENSE (C3)：
- OPENSOURCE：是否开放源代码（分为"是""否"）；
- FREE SOFTWARE：是否可免费下载、使用及分发（非商业用途，分为"是""否"）；
- TRIAL DEMO：若软件既非开源也非免费，则是否提供试用版（分为"是""否""-"；若为开源软件，则该列值为"-"）；
- HARDWARE (C4)：运行软件所需的硬件类型（商用眼动仪、网络摄像头、两者皆可、未明确）；
- PURPOSE (C5)：眼动追踪的用途或功能（如预测注视点POG、兴趣区域生成、头部运动捕捉等）；
- BENCHMARK SETUP COMPATIBILITY (C6)：软件是否与基准测试环境兼容（分为"是""否"）。
- **All Tools**：该页面展示了所有检索得到的眼动追踪软件的完整信息，未经过纳入/排除标准筛选。除了Conducting页面中的列之外，还新增了以下额外列：
- SOFTWARE END-USE CONTEXT：软件的使用场景（通用、特定）；
- SOURCE CODE：眼动追踪软件的源代码语言（Java、C++、Python、JavaScript或未明确）；
- INSTALLATION：软件的系统安装方式（可执行文件、软件即服务（Software as a Service, SaaS）、网络应用（WebApp）、编译版等）。

2. **benchmark.xlsx**：该文件收录了针对眼动追踪软件的基准测试结果，包含2个工作表：
- **准度与精度**：收录基于笛卡尔坐标系靶标开展的注视点（POG）准度与精度测试结果，各列说明如下：
- Software：被测试的眼动追踪软件；
- Test ID：测试唯一标识；
- Screenshot ID：截图唯一标识；
- Centroid POG X：眼动追踪软件估算的注视点（POG）质心X坐标；
- Centroid POG Y：眼动追踪软件估算的注视点（POG）质心Y坐标；
- Target X：目标质心X坐标；
- Target Y：目标质心Y坐标；
- Distance X (px)：注视点质心X坐标与目标X坐标的像素距离；
- Distance Y (px)：注视点质心Y坐标与目标Y坐标的像素距离；
- Euc.Distance (px)：注视点质心与目标质心之间的欧氏距离（像素单位）；
- Euc.Distance (cm)：注视点质心与目标质心之间的欧氏距离（厘米单位）；
- Average Euc.Distance (px)：单次测试中，注视点（POG）质心与目标质心的平均欧氏距离（像素单位）；
- Average euc.Distance (cm)：单次测试中，注视点（POG）质心与原点质心的平均欧氏距离（厘米单位）；
- Precision：注视点（POG）质心与目标质心平均距离的均方根（RMS）值；
- Accuracy (°)：根据测试环境（60厘米距离、眼球与屏幕目标夹角为0°）估算的准度（角度单位）；
- Accuracy per points (°)：单次测试ID下，每个采样点的估算准度（角度单位）；
- Average Accuracy per points (°)：单次测试ID下，各采样点准度的平均值（角度单位）。
- **重合度**：收录基于圆形靶标、按钮靶标及邮件发票靶标开展的注视点（POG）重合度测试结果，各列说明如下：
- Software：被测试的眼动追踪软件；
- Scenario：测试靶标对象；
- Test ID：测试唯一标识；
- Match Target：靶内匹配的注视点（POG）数量；
- Close Target：近靶匹配的注视点（POG）数量；
- Off Target：靶外匹配的注视点（POG）数量；
- Match Target + Close Target：靶内匹配与近靶匹配的注视点（POG）总数量；
- Total：总注视点（POG）数量：总计采样点数（50个）；
- %Match Target：靶内匹配的注视点（POG）占比；
- %Close Target：近靶匹配的注视点（POG）占比；
- %Off Target：靶外匹配的注视点（POG）占比；
- %Match Target + Close Target：靶内匹配与近靶匹配的注视点（POG）总占比。

3. **benchmark_tests/** 文件夹：
- **tests_instructions.txt**：测试说明手册，包含各靶标对象的测试执行流程信息；
- **tests_scenarios/**：该文件夹包含用于基准测试的四个靶标对象文件：
- circle.html
- buttons.html
- cartesian_system.html
- invoice_email.html
- **eye-tracking_software_tests/**：该文件夹按分层结构存储测试结果：
- 第一层为各待测试的眼动追踪软件文件夹（如GazeRecorder、Eyedid、Webgazer.js）；
- 第二层为各软件对应的测试靶标文件夹（buttons、circle、cartesian_system、invoice_email）；
- 第三层为每个靶标对应的测试ID文件夹；
- 每个测试ID文件夹包含两类文件：
- X_Y_Z.txt：X代表眼动追踪软件估算的靶内匹配注视点（POG）数量，Y代表近靶匹配注视点（POG）数量，Z代表靶外匹配注视点（POG）数量；
- Recording_XXXXXXXXX_YYYY.mht：Windows平台stepRecorder应用生成的用户界面日志文件，其中重合度测试靶标（按钮、圆形、邮件发票）包含30张截图，准度与精度测试靶标（笛卡尔坐标系）包含20张截图；
- 1.png至20.png：从.mht文件中导出的截图，用于准确提取注视点（POG）质心并计算准度与精度（因无法直接从.mht文件中直接获取）。
- 补充说明：GazeRecorder软件的注视点（POG）并非以常见的小点形式显示，而是以矩形包裹的眼睛图标呈现，POG_GazeRecorder.png展示了该软件估算的注视点（POG）质心。