RT-GENE: Real-Time Eye Gaze Estimation in Natural Environments

License + Attribution

This dataset is licensed under CC BY-NC-SA 4.0. Commercial usage is not permitted. If you use this dataset or the code in a scientific publication, please cite the following paper:

@inproceedings{FischerECCV2018,
author = {Tobias Fischer and Hyung Jin Chang and Yiannis Demiris},
title = "{RT-GENE: Real-Time Eye Gaze Estimation in Natural Environments}",
booktitle = {European Conference on Computer Vision},
year = {2018},
month = {September},
pages = {339--357}
}

This work was supported in part by the Samsung Global Research Outreach program, and in part by the EU Horizon 2020 Project PAL (643783-RIA).

More information can be found on the Personal Robotic Lab's website: https://www.imperial.ac.uk/personal-robotics/software/.

Overview

The dataset consists of two parts: 1) One where the eyetracking glasses were worn (and thus ground truth labels for head-pose and eye gaze are available; suffix _glasses), and 2) One with natural appearances (no eyetracking glasses are worn; suffix _noglasses). The _noglasses images were used to train subject-specific GANs, and these GANs were used to inpaint the region covered by the eyetracking glasses in the _glasses images.

There is code accompanying this dataset: https://github.com/Tobias-Fischer/rt_gene. Please use the issue tracker in the code respository if you have questions regarding the dataset.

Subjects / 3-Fold evaluation

15 participants were recorded in 17 sessions. Session 014 is a second recording of participant 002, and session 015 is a second recording of participant 005 (different days and different camera poses were used).

We used a 3-fold evaluation, with the three folds consisting of the following sessions (test on one of the groups, training with the remaining two groups):

's001', 's002', 's008', 's010'

's003', 's004', 's007', 's009'

's005', 's006', 's011', 's012', 's013'

The validation set consists of sessions 's014', 's015' and 's016'.

While the MATLAB script (prepare_dataset.m; see code repository) creates train and test images for each subject, all images were used for the evaluation (see evaluate_model.py).

Labeled dataset (sXYZ_glasses)

The file for each subject contains the following information:

label_combined.txt This is the main file containing labels. The formatting is as follows:
seq_number, [head pose: right(pos) / left(neg), up (pos) / down(neg)], [gaze: right(pos) / left(neg), up(pos) / down(neg)], timestamp

label_headpose.txt This file contains more detail about the head pose of the subject.
seq_number, [head pose translation: further(pos) / closer(neg), left(pos) / right(neg), up(pos) / down(neg)], [head pose rotation: roll right(pos) / roll left(neg), down(pos) / up(neg), rotate left(pos), rotate right(neg)], timestamp

kinect2_calibration.yaml
The kinect2_calibration.yaml file contains the camera projection matrix in ROS format (this file should not be required).

kinect2_pose.txt
The kinect2_pose.txt file contains the pose of the Kinect with respect to the motion capture system (this file should not be required).

"original" folder

The face_before_inpainting folder contains the face with a large margin to the left and right.

The mask folder contains images indicating the regions of the eyetracking glasses, aligned with the images in the face_before_inpainting folder.

The overlay folder contains images where the mask was overlaid on the face_before_inpainting images.

The face folder contains the face image extracted using MTCNN with a tighter margin.

The left and right folders contain the left and right eye image areas.

The face, left and right images were used as baseline comparison in the paper (Fig. 7 without inpainting).

"inpainted" folder

The face_after_inpainting folder contains images corresponding to the ones in the face_before_inpainting folder after applying the inpainting.

Then, the images contained in the face, left and right folders were extracted using MTCNN as above.

Unlabeled dataset (sXYZ_noglasses)

kinect2_calibration.yaml
This file contains the camera projection matrix in ROS format (this file should not be required).

kinect2_pose.txt
This file contains the pose of the Kinect with respect to the motion capture system (this file should not be required).

"face" folder
This folder contains the faces that can be used to train the GANs (without eyetracking glasses being worn).