Towards Precision in Appearance-based Gaze Estimation in the Wild

Existing appearance-based gaze estimation datasets can be broadly divided into two categories. (I) Datasets recorded in controlled environment; (II) Datasets recorded in real-world scenarios. We briefly summarized the existing datasets in Table 1. In the ’Data’ column of Table 1, the numbers represent the number of data samples available unless specified otherwise.

We can broadly classify the existing gaze estimation models into Single-Channel and Multi-channel architectures. One of the first attempts was GazeNet [32], a single channel approach where a single eye image was used as the input to an architecture based on VGG-16. GazeNet reported a 5.4^∘ mean angle error on MPIIGaze. This work was followed by Spatial-Weights CNN [31] where full face images were provided as input. Later, multi-channel approaches were proposed as an alternative to single-channel approaches. iTracker [10] was one of the first multi-channel architecture which used left eye image, right eye image, face crop image and face grid information as inputs. Multi-Region Dilated-Net proposed by Chen and Shi [3] employed dilated convolutions in their proposed model and used both eye images along with face image as inputs. This approach also reported 4.8^∘ mean angle error as [31] did on MPIIFaceGaze. Cheng et al. proposed FAR-Net [5] which utilized the asymmetry between two eyes of same person to obtain gaze estimates. In this work, they generated confidence scores for the gaze estimates obtained from two eye images to choose the more accurate prediction. Murthy et al. [18] proposed an approach for gaze estimation based on difference mechanism and reported 4.3^∘ and 8.44^∘ accuracy on MPIIFaceGaze and RT-GENE datasets respectively. The difference mechanism utilized the absolute difference of the feature vectors computed from both eye images for gaze estimation. Cheng et al. [4] proposed a coarse to fine approach for building a gaze estimation method and reported 4.14^∘ and 5.3^∘ accuracy on MPIIFaceGaze and EYEDIAP datasets respectively. Recently, Murthy and Biswas proposed AGE-Net [16] which utilized feature-level attention mechanism to obtain a 4.09^∘ and 7.44^∘ error on MPIIFaceGaze and RT-GENE datasets respectively.

In summary, we observed that only two appearance-based gaze estimation datasets were collected in real world scenarios under interactive setting. Among them, MPIIFaceGaze [31] was recorded under static settings with limited head pose distribution and GazeCapture [10] contained limited gaze direction distribution. Further, no gaze estimation model was evaluated for its precision.

Once a recording session was initiated by the participant, our custom software displayed a red dot within two concentric circles, dubbed as the Target, at a random location on screen. The recording procedure was divided into two phases, selection phase and fixation phase. In the selection phase, the participant had to look at the Target and click on it. We monitored the distance between the Target and mouse click position to verify whether participant selected it correctly. Once the user properly selected the Target, the Target image shrunk to a minimum size. This served as an affirmative feedback to the participant for their click action and also to obtain more precise ground truth point. Once the Target reached to a minimum size, the fixation phase could be initiated by the participant by pressing space bar on keyboard. During the Fixation phase, participants were asked to fixate on the Target while varying their head pose or facial appearance or position with respect to screen. Moreover, they were asked to be natural during fixation phase and the only constraint was to maintain their gaze at the Target point. Our data recording happened in two phases spanning over a year. In the first phase, we allowed participants to start and stop the fixation at their own pace and in second phase, we terminated each fixation after a predefined time of 7 seconds. We turned the application background to black once the fixation starts and ensured that no other screen element including cursor got displayed during the fixation phase to minimize on-screen distraction. They were also given the option to terminate a fixation before the predefined duration by pressing ’Esc’ key on keyboard in the event of any distraction or eye blinks. Participants got acquainted with the user interface and the recording software before the actual data recordings. Participants were located at different geographic locations and we instructed them to record in both indoor and outdoor locations.

The number of sessions recorded by a single participant varied from 1 session to 23 sessions. Each session was predefined with 30 fixations, but many sessions had lesser fixations due to fixation terminations or session terminated by the participant. We recorded timestamps of participant’s click on the Target, space bar press and ’Esc’ key press during the recording session. We observed from the recordings that all participants were static till the moment they began their fixation and they stopped orienting the head once the fixation was terminated. We believed that the predefined selection and fixation phase design for data recording incorporated this head movement pattern and it allowed us to verify the attention of participant during the fixation phase. The mean duration over all fixation was 6.93 $\pm$ 1.26 sec. From our fixation recordings, we sampled at higher frequency (6 Hz) than the procedure reported for EYEDIAP in [5, 31, 4] (2 Hz) since the dynamics in terms of facial appearances and head pose in our dataset was higher. Further, EYEDIAP dataset contains both static and dynamic sessions whereas all fixations in the proposed dataset are dynamic by design. We obtained landmarks for sampled frames using OpenFace[1]. We followed the procedure described in [29] for image and gaze data normalization. The image normalization process cancels out the roll component of head orientation and places the image at a fixed distance from a virtual camera. Further, we transformed the PoG to a normalized space using extrinsic calibration [25] of the laptop cameras, head pose and the head rotation matrix of the participant.

Participants (17 Male and 11 Female; 16 with spectacles; Age: [19-56] years) belong to different states of India. We recorded 974 minutes of video recordings ( $\approx 1.7$ Million images) over 328 sessions. The proposed dataset duration was larger than EYEDIAP (237 minutes) and data was collected in real-world conditions rather than in a lab setting. All sampled frames were manually verified and the frames with eyes closed or user distraction from Target were discarded. It was observed that the distraction can occur due to external noise while recording outdoors or the ambient environmental factors like wind or dust. Further, samples with confidence score less than 0.8 while detecting landmarks were discarded resulting in a final evaluation subset of 300,961 samples, larger than MPIIGaze [32] and Gaze360 [9]. It may be noted that Gaze360 provides data at higher sampling rate of 8 Hz, yet contain less number of samples than the proposed dataset. A few representative images from the evaluation set were presented in Figure 1.

We studied the performance of various existing gaze estimation models on the PARKS-Gaze dataset. We chose Dilated-Net [3], I2D-Net [18] and AGE-Net [16] since these models represent three levels of performance (4.8^∘, 4.3^∘ and 4.09^∘ error respectively) on challenging MPIIFaceGaze dataset. We observed that other existing gaze estimation models [4, 6, 5] reported similar performance as these methods. For our with-in dataset evaluation, we have separated our 28 participants into 3 groups, Train, Validation and Test splits with 18, 4 and 6 participants in each group respectively. We ensured similar gaze and head pose distribution across three groups.

As denoted in Table 2, AGE-Net achieved state-of-the-art performance on MPIIFaceGaze, EYEDIAP, RT-GENE and on the proposed PARKS-Gaze datasets. AGE-Net recorded a mean angle error of 5.17^∘ across the 6 test participants of PARKS-Gaze. Since image resolution in both MPIIFaceGaze and PARKS-Gaze datasets is same, we posit that the higher with-in dataset validation error indicate diverse conditions and more number of challenging samples in the proposed dataset than MPIIFaceGaze. Since AGE-Net achieved superior performance across all datasets, we considered it for further evaluations.

We conducted pair-wise cross-dataset evaluations using MPIIFaceGaze, RT-GENE, EYEDIAP, GazeCapture, ETH-XGaze and the proposed PARKS-Gaze datasets. We followed the normalization procedure described in [29] for all the above mentioned datasets. We used 960mm as the focal length of the virtual camera. The distance between the virtual camera and the image center for eye and face images is considered to be 600mm and 1000mm respectively. All the warped images are converted into grayscale and histogram equalized. Since authors of RT-Gene [6] provided the normalized images, we resized and converted them to grayscale as per the model’s requirements. In the case of EYEDIAP, we considered 14 participants with screen targets. We followed similar procedure followed in [5, 31] and sampled 1 frame for every 15 frames from VGA videos. We also considered the available frame annotations to eliminate frames with eyes closed or participant not looking at the Target from our evaluation. Since GazeCapture do not have 3D gaze annotations, we utilized the normalized gaze annotations and head pose information provided by [20]. We considered 1176, 50 and 140 participants for training, validation and test splits. For the ETH-XGaze dataset, we obtained landmarks using [2] and obtained normalized eye images from the given face images.

We summarized the results of our cross-dataset evaluation in Table 3. We ranked training datasets based on the mean angle error for each test dataset and assigned average ranking using the method described in [28]. We observed training on PARKS-Gaze dataset leads to lowest test errors for RT-GENE and GazeCapture datasets and it obtained the best rank (1.6) followed by GazeCapture (1.8) among the 6 training datasets. Further, training on GazeCapture achieved lowest test error when tested on MPIIFaceGaze, ETH-XGaze and PARKS-Gaze datasets even though its gaze distribution is smaller than the test datasets.

ETH-XGaze dataset reported 6.63^∘, 7.97^∘ and 11.65^∘ test error on MPIIFaceGaze, GazeCapture and PARKS-Gaze dataset respectively. Since ETH-XGaze contained large number of participants and larger head angles, higher error on the proposed dataset than the MPIIFaceGaze and GazeCapture further corroborates our argument that PARKS-Gaze is more challenging than the other two datasets. Further, it reported large test errors on EYEDIAP and RT-GENE possibly due to the difference in image resolution between source and target datasets. We observed high errors as reported in [28] when tested on ETH-XGaze due to the domain gap between other datasets and ETH-XGaze in terms of head pose and gaze angle distribution. It may be noted that the results demonstrated here in Table 3 are superior to the ones obtained using the earlier version of the dataset [13].

Our literature review revealed that the evaluation of gaze estimation model’s performance is predominantly confined to gaze angle errors. Such measurements do not inform us about the actual usability of such systems for interactive gaze-controlled applications. We undertook performance analysis of gaze estimation models at the PoG level investigating both accuracy and precision error. We performed these experiments in two fold. First, we performed precision analysis on the three models that we considered for our within-dataset validation. Second, we analyzed the effect of training dataset on precision error.

We defined the following experimental procedure to analyse precision error on an unseen participant. As described during within-dataset evaluation, we split our dataset into 3 groups with 18, 4 and 6 participants. We name these splits as Train, Eval1, Eval2. We trained each model twice with Train split using Eval1 and Eval2 as the validation data and test data alternatively. This allowed us to investigate precision on a total of 10 unseen participants with 2504 fixations. We measured PoG error corresponding to each frame in a fixation using euclidean distance between the ground truth and the prediction in screen coordinate system measured in millimeters. We measured precision error using the standard deviation of all PoG predictions corresponding to each fixation. In Figure 5, we presented the mean fixation error, an average value of PoG errors over all fixations of all 10 participants. Similarly, we presented Mean Precision_X and Mean Precision_Y which is an average value of all precision values computed over fixations for all participants in X and Y directions. We also illustrated the mean angle error of each model across all 10 test participants using a line plot. Two paired t-tests revealed that AGE-Net obtained statistically significant improvement over I2D-Net in precision in both X (t[9] = 3.02, p = 0.007) and Y (t[9] = 5.34, p = 2E-4) dimensions. Further, AGE-Net reported significant improvement over Dilated-Net in terms of precision in X-dimension (t[9] = 3.40, p = 0.003).

The cross-dataset evaluation reported in Section 4.2 discusses only the accuracy of the model on an unseen participant from a different dataset. Here, we investigate the effect of training dataset on precision error under similar setting. For this purpose, we consider AGE-Net model as it reported superior precision over the other two models during within-dataset precision evaluation.

We used Eval1 and Eval2 splits of the proposed PARKS-Gaze dataset and Test split of GazeCapture dataset as test datasets for precision analysis. We discarded fixations from GazeCapture test split which have only one corresponding frame. The number of frames for remaining fixations ranged from 2 to 15. Since authors of GazeCapture did not provide extrinsic calibration parameters, R, T of the camera with respect to the display, we attempted to generate them using the available information of screen points of the fixation and their corresponding representation in camera coordinate system. We were able to obtain extrinsic parameters for 23026 fixations. The R, T are essential to map gaze angle predictions to screen coordinate system.

On the Test split of PARKS-Gaze dataset, training on MPIIFaceGaze resulted in a mean precision error of 57.2 $\pm$ 33 mm and 33.4 $\pm$ 14 mm while training on GazeCapture resulted in a mean precision error of 37.3 $\pm$ 16 mm and 32.6 $\pm$ 11 mm in X and Y directions respectively. Figure 6 illustrates the precision error obtained using MPIIFaceGaze, GazeCapture datasets. We further included results obtained when the model was trained on the Train split of PARKS-Gaze dataset here for the sake of comparison. Two paired t-tests revealed that during this evaluation case, GazeCapture recorded statistically significantly lower precision error when compared to MPIIFaceGaze in X direction (X - t[9] = 3.4, p = 4E-3) while no significant difference was observed for precision error in Y direction.

Next, AGE-Net model trained on the Train split of GazeCapture dataset reported a mean precision error of 8.2 $\pm$ 4.5 mm and 7.99 $\pm$ 3.5 mm in X and Y directions respectively when tested on its Test split. On the Test split of GazeCapture dataset, training on MPIIFaceGaze resulted in a mean precision error of 13.2 $\pm$ 6.5 mm and 10.4 $\pm$ 3.8 mm while training on the proposed PARKS-Gaze resulted in a mean precision error of 9.2 $\pm$ 4.2 mm and 8.07$\pm$2.8 mm in X and Y directions respectively. Two paired t-tests revealed that the proposed PARKS-Gaze dataset recorded statistically significantly lower precision error than MPIIFaceGaze on both X direction (X - t[139] = 14.5, p = 1E-29) and Y direction (Y - t[139] = 16.56, p = 6E-35). Hence, MPIIFaceGaze dataset is outperformed by both PARKS-Gaze and GazeCapture in two different scenarios in terms of precision errors.

It is worth noting that we observed no significant difference between precision errors in Y-direction obtained using PARKS-Gaze dataset and GazeCapture dataset when tested on Test split of GazeCapture dataset. Further, it may be noted that the combined mean precision error of PARKS-Gaze under cross-dataset setting (12.33 $\pm$ 4.9 mm) is only 0.8 mm lower than the precision-error under within-dataset setting using GazeCapture (11.52 $\pm$ 5.6 mm). GazeCapture training dataset has 1176 participants whereas PARKS-Gaze training split contained only 18 participants, yet the obtained precision errors under challenging cross-dataset setting are on-par to the errors obtained during a within-dataset evaluation. This indicates strong generalizability of the proposed dataset on unseen participants and its ability in generating precise gaze estimates.

The mean precision error for fixations in Test split of PARKS-Gaze with head pose variation $\leq 5^{\circ}$ was observed to be 10mm and 9.8mm in X and Y directions. This represents the scenarios where a person might be dwelling on a button with an intention to select and these values decide the design choices while building gaze-controlled interfaces. A similar analysis on GazeCapture resulted in a precision error of 13.5 mm, 14.8mm and on MPIIFaceGaze resulted in 15.7 mm, 13.7 mm precision error in X and Y directions. This indicate that the proposed dataset achieved a lower precision error on unseen participants even under limited head pose variation than existing in-the-wild datasets. Appearance-based gaze estimation is a function of many factors and the role of these factors in conjunction on the precision is not studied.