Bayesian Eye Tracking

Eye tracking is to identify people’s focus of attention(or line of sight) in the 3D space or on the 2D screen. Applications of eye tracking range from marketing, psychology, user behavior research to gaming, human-computer interaction, medical diagnosis, etc.

Recent gaze estimation methods can be divided into two categories: 1) learning-based and 2) model-based. Learning-based methods [13, 16, 25, 36, 37, 14] aim at learning a mapping function from the input image to eye gaze. The most popular mapping function nowadays is the convolutional neural network and its variants. Despite their impressive performance on benchmark datasets, the major issue with learning-based methods is that they cannot generalize well and that they require significant amount training data with eye gaze annotations to perform well. Accurate eye gaze annotation is a tedious, expensive, and time-consuming process and it often requires specialized equipment [32] . Model-based methods leverage on a geometric model representing the anatomical structure of eye to estimate eye gaze. The idea is to first detect 2D eye or facial landmarks, then reconstruct the 3D location of eyeball center and pupil center from 2D observations. Finally, 3D eye gaze can be estimated by connecting the eyeball center and pupil center. Early efforts were focused on designing new geometric models [1, 11, 12, 10, 15, 20, 2, 33, 35, 26, 27] to better recover 3D information from 2D observations. Recently, researchers are more interested in improving the accuracy of eye/facial landmark detection [6, 18, 28, 19] with the powerful deep learning tools. While training-free and generalizabale, the model-based approach often cannot produce accurate eye gaze estimation due to inaccuracies with eye landmark detection, in particular for real world eye tracking, unless special hardware (e.g IR light) is used.

To overcome these limitations, we introduce a Bayesian framework for model-based eye tracking without explicit eye landmark detection. As shown in Fig. 1, the proposed method consists of a Bayesian Convolutional Neural Network (BCNN) module which captures the relationship between eye appearance and eye landmarks, and a geometric model which relates eye landmarks to eye gaze. The two modules are jointly formulated through a Bayesian probabilistic framework, and eye gaze can be effectively estimated through a Bayesian inference. With the Bayesian inference, eye landmark predictions on an input eye image can be performed by an ensemble of BCNN models drawn from their posterior distributions. The landmarks generated by each BCNN model are then used to estimate eye gaze for the input image, yielding a distribution of eye gaze. By modeling the landmark distribution and gaze distribution, the proposed model can generalize to different subjects, head poses or new environments and yield robust performance even under noisy images. Furthermore, our model also associates a level of uncertainty with its gaze prediction. Uncertainty is essential to understand what deep networks cannot do and avoid over-confident predictions. Finally, by feeding the uncertainty information from current stage to next stage with the cascade architecture, we can further improve the model performance. To summarize, we make following novel contributions:

• •

  We introduce a unified Bayesian framework that combines a model-based eye tracking with a Bayesian convolutional neural network to allow robust and generalizable model-based eye tracking .

• •

  With Bayesian inference, our model avoids explicit landmark detection and requires no eye gaze annotations and no model training.

• •

  The model produces not only eye gaze predictions but also their uncertainties, which can be incorporated into a cascade framework to progressively improve gaze estimation performance.

In this section, we focus on reviewing recent work on model-based eye gaze tracking, in particular those aimed at improving eye feature detection and eye tracking generalization performance. In [27], the authors proposed to estimate the 3D eye gaze given 2D facial landmarks. They introduce a 3D eye-face deformable model that relates the 3D eyeball center with 3D rigid facial landmarks. Given the detected 2D facial landmarks and the eye-face model, they can effectively recover the 3D eyeball center and obtain the final gaze direction. In [19], the authors proposed to recover the gaze direction with eye region landmarks. They first used a deep network (Hourglass) to detect eye region landmarks. Next they leveraged on the proposed geometric eye model to estimate the eye gaze direction given the detected eye landmarks. Both methods require accurate facial and eye region landmark detections to perform well. In [21], authors focused on robust eye feature detection in pervasive scenarios. They introduced the Pupil Reconstructor(PuRe), which is mainly based on an edge segment selection. Their method also includes the confidence measure for the detected pupil. But PuRe fails to work on low-resolution eye images. Sun, et al. [24] proposed a gaze estimation system that consists of three modules: gaze feature extraction, system calibration, and gaze estimation. The gaze features are fed into the gaze estimation module after the calibration procedure. During the gaze feature extraction, a parameterized iris model is used to locate iris center, which is robust against low-qulity eye images and large head movement. Wood, et al. [31] constructed a 3D deformabel dense eye model from the head scans. Their model includes shape and texture variations of facial eye region (eyebrow, eyelid, part of the nose) and eyeball (iris, sclera). By fitting the whole eye model instead of fitting landmarks, they can get the 3D position of eyeball center and pupil center simultaneously. But during fitting, they still used landmarks as regularization and set high weights to it. And their model is not robust when facing real world scenario, especially when occlusion occurs. Wang et al. [29], proposed to leverage Bayesian inference during the adversarial learning to increase generalization for appearance-based gaze estimation. However, as a learning based instead of model-based, their method required a large amount of labeled training data under different variations and a complex training procedure; it hence cannot generalize well beyond the training data.

Given data $\mathbf{D}=\{\mathbf{I}_{i},\mathbf{z}_{i}\}_{i=1}^{N}$, where $\mathbf{I}_{i}$ and $\mathbf{z}_{i}$ represent the image and corresponding landmarks, model prior $\boldsymbol{\alpha}$ and geometric model parameter $\boldsymbol{\theta}$, our goal is to estimate the gaze $\mathbf{g}_{t}$ for a testing image $\mathbf{I}_{t}$. We want to emphasize here that $\mathbf{D}$ contains no eye gaze annotations and that its landmarks are not manually annotated but are generated by a third party landmark detector.

We first introduce the deterministic model formulation, then we discuss our extension to the probabilistic formulation.

Following the order in Eq. (4), we first discuss how to obtain the landmark distribution $p(\mathbf{z}_{t}|\mathbf{I}_{t},\mathbf{w})$ for a testing image in Sec. 4.1. This is then followed by discussing the parameter posterior $p(\mathbf{w}|\mathbf{D},\boldsymbol{\alpha})$ in Sec. 4.2 and then the model-based gaze estimation $p(\mathbf{g}_{t}|\mathbf{z}_{t},\theta)$ in Sec. 4.3. Finally, given these three terms, we summarize in Sec. 4.4 how we can approximately solve for Eq. 4.

As the output of BCNN are the probability distribution of the landmark points, we can use them to improve landmark detection. Moreover, several studies show that the cascade framework can yield better landmark predictions [30]. We adopt the same idea of cascade framework with a modification that we incorporate the uncertainty information (Fig. 5). Compared to [30], our method explicitly models the landmark distribution as a Gaussian distribution in Eq. (5), and predicts the mean and variance of the distribution, while [30] predicts a deterministic landmark heatmap. In addition, [30] uses artificially generated heatmap with fixed variance as supervision. Their fixed variance needs to be tuned carefully while our method only requires landmark locations as groundtruth without additional assumptions.

As can be seen in Fig. 4 (c), the $m$ samples of $\mathbf{w}$ produce $m$ posterior distributions of landmarks for the same image $\mathbf{I}$, from which we can compute both the epistemic and aleatoric uncertainties:

where $\boldsymbol{\mu}_{(1)}$ represents the average landmark predictions from the $m$ posterior samples $\mathbf{w}^{i}$, while $\boldsymbol{\Sigma}_{(1)}$ represents the total uncertainty about the prediction. Total uncertainty consists of the epistemic uncertainty term and the aleatoric uncertainty term. Epistemic uncertainty captures the covariance of the mean of the $m$ predictions. Intuitively, if the $m$ predictions are close to each other, we are more confident in the prediction, otherwise if the $m$ predictions are different from each other, we have high uncertainty about the prediction. Aleatoric uncertainty is the mean of the predicted covariance. It encodes the uncertainty about our input $\mathbf{I}$. It is high if $\mathbf{I}$ is different from the training data distribution (e.g., noise, outlier).

Since total uncertainty captures the quality of the landmark predictions, we would like to incorporate uncertainty information to improve the performance. We propose to encode the total uncertainty into a probability map (same size as the feature map extracted from convolutional layers). The probability map encodes the probability of the landmark being at a particular pixel location.

To this end, we carefully design a probability map layer $\boldsymbol{\phi}(\cdot)$, that converts $\boldsymbol{\mu}_{(1)}$ and $\boldsymbol{\Sigma}_{(1)}$ to a probability map $\mathbf{p}=\boldsymbol{\phi}(\boldsymbol{\mu}_{(1)},\boldsymbol{\Sigma}_{(1)})$. The idea is to apply a Gaussian filter at location $\boldsymbol{\mu}_{(1)}$ with the total uncertainty $\boldsymbol{\Sigma}_{(1)}$ as covariance matrix. Let us consider the case for one landmark. Assume image is of size $h\times w$, $\boldsymbol{\mu}_{(1)}=(x,y)$, $diag(\boldsymbol{\Sigma}_{(1)})=(\sigma_{x},\sigma_{y})$ (we assume independent dimensions of $\mathbf{z}$ and only use the diagonal entries) . We first create meshgrids $\mathbf{X}$ and $\mathbf{Y}$ with size $h\times w$, then the value of grid $(x,y)$ can be computed as follows:

To better understand how Eq. (5) works, we provide several examples in Fig. 6. Intuitively, if we have high uncertainty of the landmark prediction, the probability map will spread out (first row of Fig. 6), and encourage the network to search around the neighborhood to refine the prediction. Similarly, network is encouraged to stay at current locations if we are confident on the prediction (peak regions as in second row of Fig. 6). The complete c-BCNN is illustrated in Fig. 5, the $Conv_{s}$ blocks are shared for Stage $\geq 2$. The probability map and the feature map is concatenated and fed to the next BCNN block. With multiple stages, we can still use Alg. 1 to perform gaze estimation. The difference is that the $\boldsymbol{\mu}(\mathbf{I};\mathbf{w})$ and $\boldsymbol{\Sigma}(\mathbf{I};\mathbf{w})$ in the log-likelihood (Eq. (5)) now is parameterized by the multi-stage CNN in Fig. 5, where $\mathbf{w}=\{\mathbf{w}_{(1)},...,\mathbf{w}_{(k)}\}$ represents the parameters for all $k$ stages.

We consider four benchmark datasets for our evaluation. 1) EyeDiap [7]: The dataset contains different sessions (e.g. HD camera VS VGA camera, discrete target VS floating target, small head motion VS large head motion). We use images captured by VGA camera with different head poses. 2) Columbia [23]: The dataset has high quality images $5181\times 3456$ from $55$ subjects. The $5880$ images are all used for evaluation. 3) UT [34]: The dataset contains $50$ subjects each with $8$ different head poses and $160$ gaze directions. All images are used for evaluations. 4) MPIIGaze [37]: The dataset has $15$ subjects and there are head pose variations and lighting variations. We follow the same evaluation sittings to test on the evaluation subset from the dataset.

The eye-region model is applied to the four datasets, while the eye-face model is only applied to EyeDiap and ColumbiaGaze since they provide with full face images and camera parameters (required by eye-face model). Gaze estimation error is measured by the angular error in degree. Notice that these benchmark datasets do not provide landmark labels. For training data, we directly apply a third-party landmark detector [8, 9] to generate landmarks (we use $11$ facial landmarks as in [27] and $18$ eye landmarks as in [19]). In spite of the inaccuracies with the landmark detector, we can also achieve comparable performance. For the cascade framework, we find that $6$ stages work best for facial landmarks and $3$ stages are sufficient for eye landmarks detection. All network inputs (face image/eye image) are resized to $184\times 184$. For the convolutional blocks in each stage, we adopt a 4-layer U-Net like structure (Fig. 5). The feature map size for the 4-layers are $92\times 92$, $46\times 46$, $46\times 46$ and $92\times 92$. For the geometric model in [27], we use the average 3D deformable eye-face model. We also use human-average eye related parameters (e.g. eyeball radius is 12 mm, kappa angle is 5.0 and 1.2 degrees) for both eye-face model and eye-region model. For the Bayesian inference, we use $m=50$ parameter samples and $n=50$ landmark samples.

We introduce a novel Bayesian framework to improve the generalization and robustness of the model-based eye tracking. It consists of a BCNN, which captures the probabilistic relationship between appearance and landmarks, and a geometric eye model to relate landmarks to gaze. The two components are unified by a Bayesian framework to avoid point estimation and the explicit landmark predictions. We further extend the single-stage model to multi-stage and incorporate uncertainty information to progressively improve the model performance. Experiments demonstrate better generalization and robustness with Bayesian inference and the cascade framework. Furthermore, our method outperforms SoA model-based eye tracking methods on challenging datasets, further demonstrating its generalization power.