Benchmark Webcam Eye-Tracking Software

These data correspond to the industrial systematic review and benchmark test set related to the paper A Benchmark Study of Webcam Eye Tracking Software in Robotic Process Automation.

The industrial systematic review is an approach adapted from Kitchenham's method that gathers all the eye tracking tools obtained after running a search in search engines, entering defined queries, and filtering by inclusion/exclusion criteria that we define in our planning. More precisely, this review includes all the different results obtained from the search queries (Eye tracking or Gaze tracking or Eye tracker) and (software or app or tool or github) entered in the search engines DuckDuckgo, Bing and Google.

After the search, the eye tracking software results will be filtered according to the exclusion/inclusion criteria (C):

·       C1: The software must be available and documented or supported.

·       C2: The software must work on desktop computers or laptops, excluding mobile devices.

·       C3: The software should be open-source and/or free software, and if it is not, a demo version should be available even if it is for a trial period.

·       C4: The software must provide the option to use the built-in/native camera of a PC or webcam as hardware for eye tracking. It is excluded head-mounted devices, Virtual Reality glasses and commercial eye tracker hardware.

·       C5: The software must provide Eye tracking as software, not as a feature or extension of a service unrelated about specifically predicting the POG as functionality.

·       C6: The software must be installable in the established benchmark test setup.

The software that passes the inclusion/exclusion criteria are those evaluated in a benchmark.

 For this evaluation, there are 4 different objects where eye tracking software tests are performed. These objects are called Circle, Buttons, Cartesian system and Email Invoice and differ from each other in the elements and location of the targets for testing each eye tracking software.

The results obtained from tests objects are strategically differentiated into two distinct categories to becnhmark eye tracking software according to Accuracy, Precision and POG Coincidence metrics:

·       Accuracy and precision. Measures directly the accuracy and precision of the eye tracking software in the Cartesian system object.

        o   Accuracy. It is quantified as the angular error between the point representing the centroid of the estimated POG from the eye tracking software and the point representing the centroid of the target, at which the subject is gazing.

        o   Precision. It is calculated using the root mean square (RMS) of the sampled points obtained from the screenshots of one test session.

·       POG coincidence. Classifies where the estimated POG is in each screenshot. This category applies to Circle, Buttons and Invoice email objects.

        o    Match on target (green circle with a diameter of 1cm). The estimated POG centroid by the eye tracking software in the screenshot is inside the green circle area or touches its circumference or limit.

        o   Match in a close target (blue circle with a radius of 2cm that contains the green circle (target) inscribed within it. The estimated POG centroid by the eye tracking software in the screenshot is inside the blue circle area or touches its circumference or limit.

        o   Off-target. The estimated POG centroid by the eye tracking software in the screenshot is outside the target limit

The structure of the data packages is organized as follows:

SLR_eye_tracking_software.xlsx: This .xlsx file corresponds entirely to the entire industrial systematic review on eye tracking software. It consists of 3 sheets:

SLR: This encapsulates the eye tracking software acquired through search queries inputted across diverse search engines. It is composed of 4 columns.

SOFTWARE: Web platform or application that provides the eye tracking software.

SOFTWARE URL: URL to the repository or host page of the software.

SEARCH URL: URL of the page mentioning the software

QUERY SEARCH: Keywords used for the search and entered in the search engines.

Conducting: This page shows the classification of eye tracking software according to the established inclusion/exclusion criteria, where each column corresponds to one of the criteria. Software that does not pass the successive criteria is discarded. Columns are as follows:

AVAILABILITY; DOCUMENTATION AND SUPPORT (C1): Software that is downloadable and supported for viable use (Available, Not Available). Documentation and/or support about the software setup, run and outputs (Yes, No).

PLATFORM (C2): Type of device and operating system where the sofware is run (PC: Windows, Ubuntu, MacOs, ...; Mobile: iOs, Android ...; Multiplatform).

LICENSE (C3):

OPENSOURCE: Public access to the source code (Yes, No).

FREE SOFTWARE: It can be downloaded, used and distributed free of charge for non-commercial purposes (Yes, No).

TRIAL DEMO: Availability of trial version if the software is neither open source nor free (Yes, No, -). *If it is opensource, trial demo value is “-“.

HARDWARE (C4): Hardware required to run the software (Commercial, Webcam, Both, It is not determined).

PURPOSE (C5): Eye tracking purposes or functionalities (Predict POG, AOI generation, Head motion capture, ...).

BENCHMARK SETUP COMPATIBILITY (C6): The software shows compatibility with the benchmark setup (Yes, No).

All Tools: On this page, all the information obtained from each of the eye tracking software is displayed, without discarding based on the inclusion/exclusion criteria. In addition to the columns described on the conducting page, additional columns with extra information are included.

SOFTWARE END-USE CONTEXT: Purpose of use of the software (Generic, Specific).

SOURCE CODE: Eye tracker source code language (Java, C++, Python, JavaScript or it is not determined).

INSTALLATION: Installation of the software in the system (Executable, SaaS, WebApp, Compiled, ...).

benchmark.xlsx: This file represents the results of the benchmark carried out on eye tracking software. It consists of two sheets.

         ·       Accuracy + Precision: It corresponds to the results obtained from the POG accuracy and precision tests conducted based on the Cartesian system object. Each column represents the following data:

                       o   Software: Eye tracking software tested.

                       o   Test ID: Test identification.

                       o   Screenshot ID: Screenshot identification.

                       o   Centroid POG X: X-coordinate of the centroid of the (POG) estimated by the eye tracking software.

                       o   Centroid POG Y: Y-coordinate of the centroid of the (POG) estimated by the eye tracking software.

                       o   Target X: X-coordinate of the centroid of the target.

                       o   Target Y: Y-coordinate of the centroid of the target.

                       o   Distance X (px): Distance from the Centroid POG X to Target X (in pixels).

                       o   Distance Y (px): Distance from the Centroid POG Y to Target Y (in pixels).

                       o   Euc.Distance (px): Euclidean distance (in pixels) from the centroid of the POG to the centroid of the target point.

                       o   Euc.Distance (cm): Euclidean distance (in centimeters) from the centroid of the POG to the centroid of the target point.

                       o   Average Euc.Distance (px): Average Euclidean distance (in pixels) from the centroid of the Point of Gaze (POG) to the centroid of the target point, referred to the same test.

                       o   Average euc.Distance (cm): Average Euclidean distance (in centimeters) from the centroid of the Point of Gaze (POG) to the centroid of the origin point, referred to the same test.

                       o   Precision: Value of the Root Mean Square (RMS) of the average distance from the centroid of the Point of Gaze (POG) to the centroid of the target point.

                       o   Accuracy (º): Accuracy estimated (in degrees) according to the setup (60 centimeters and 0º eyes to screen target distance).

                       o   Accuracy per points (º): Accuracy estimated (in degrees) per points referred to the same Test ID

                       o   Average Accuracy per points (º): Average Accuracy (in degrees) per points referred to the same Test ID.

         ·       Coincidence: It corresponds to the results of the benchmark from the POG coincidence tests conducted based on Circle, Buttons and Invoice Email objects. Each column represents the following data:

                       o   Software: Eye tracking software tested.

                       o   Scenario: Object tested.

                       o   Test ID: Test identification.

                       o   Match Target: Number of Match target POG.

                       o   Close Target: Number of close target POG

                       o   Off Target: Number of Off target POG.

                       o   Match Target + Close Target: Sum of the match target POG and Close Target POG.

                       o   Total: Number of total POG: Total number of recounted points (50).

                       o   %Match Target: Percentage of match target POG.

                       o   %Close Target: Percentage of close target POG.

                       o   %Off Target: Percentage of off target POG.

                       o   %Match Target + Close Target: Sum of the match target POG and close target POG percentage.

benchmark_tests/

          ·  tests_instructions.txt

                       It is an instruction manual containing information on how the tests are conducted in each object.

           ·  tests_scenarios/

                       This folder contains the four objects designed for the benchmark evaluation.

                       o   circle.html

                       o   buttons.html

                       o   cartesian_system.html

                       o   invoice_email.html

            ·  eye-tracking_software_tests/

                        o   GazeRecorder/

                                     -  buttons/

                                                 ·   T1/

                                                              o   X_Y_Z.txt

                                                              o   Recording_XXXXXXXXX_YYYY.mht

                                                  ·   T2/

                                                              o   X_Y_Z.txt

                                                              o   Recording_XXXXXXXXX_YYYY.mht

                                                   ·   T3/

                                                              o   X_Y_Z.txt

                                                              o   Recording_XXXXXXXXX_YYYY.mht

                                       -  circle/…

                                       -  invoice_email/…

                                       -  cartesian_system/

                                                    ·   T1/

                                                               o   Recording_XXXXXXXXX_YYYY.mht

                                                               o   1.png

                                                               o   2.png

                                                               o   …

                                                               o   20.png

                                                       ·   T2/

                                                               o   Recording_XXXXXXXXX_YYYY.mht

                                                               o   1.png

                                                               o   2.png

                                                               o   …

                                                               o   20.png

                                                               -  (*) POG_GazeRecorder.png

The folder eye tracking_software_tests contains the results of the tests in a stratified manner. At the first level, there are folders with data belonging to the respective eye tracking software to be tested (GazeRecorder, Eyedid, and Webgazer.js). At the next level, corresponding to each software, there would be folders for the test objects  (buttons, circle, cartesian_system, and invoice_email). Within each object test folder, we find folders corresponding to the Test ID. Within each Test ID, there are two files:

          ·    X_Y_Z.txt.

                             o   X represents the number of match target POG estimated by the eye tracking software.

                             o   Y represents the number of of close target POG estimated by the eye tracking software.

                             o   Z represents the number of off-target POG estimated by the eye tracking software.

            ·    Recording_XXXXXXXXX_YYYY.mht.

                             o   It is the User Interface log generated by the stepRecorder application for Windows. It contains 30 screenshots about the POG estimation for the coincidence tests objects (Buttons, Circle and Invoice Email) and 20 screenshots for the accuracy and precision test object (Cartesian System).

            ·    1.png to 20.png

                             o   These screenshots are exported from the .mht file in order to obtain the centroid of the Point of Gaze (POG) and calculate accuracy and precision accurately, as it is not possible directly from the .mht file.

(*) The Point of Gaze (POG) in GazeRecorder is not displayed as a small dot, as is common and as it happens in EyeDid and Webgazer.js, but rather as an eye encapsulated in a rectangle. POG_GazeRecorder.png shows the centroid for the POG estimated by that eye tracking software.