Gaze Perception in Humans and CNN-Based Model

Two hundred participants (at least 18 years old) were recruited with a Human Intelligence Tasks(HIT) posted on Amazon Mechanical Turk. One hundred participants were assigned to the target-present condition, and the other hundred participants were assigned to the target-absent condition. All participants were asked to make an explicit perceptual estimation of the locus of gaze by clicking on the image where they thought all the gaze-orienting individuals were looking at. Sixty images extracted from 60 videos were presented in a random order to each subject (18 images had one gaze-orienting person, 18 images had two gaze-orienting people, 24 images had three gaze-orienting people). Subjects had unlimited viewing time and were able to adjust their selections before moving to the next image. Importantly, each participant only viewed one version of the image from the same video to prevent the memory from interfering with their judgment.

We used a state-of-the-art model that was pre-trained on predicting attended targets in videos (Chong et al., 2020). The model requires the whole image of the scene and a cropped region of one person’s head as input. These images are passed to two convolution paths to extract features. For the head region, the model also creates an attention map. After combining scene features, head features, and the attention map, the model uses an Encoder, Conv-LSTM, and Deconv layer to produce a probability map to indicate the predicted attended location in the image. Note that the current version of the model only makes predictions on a single gaze-orienting person at a time.

For human estimations of the locus of attention, we used the mean x,y coordinates from all participants’ location selections. For the model’s estimation, we used the peak location of the probability map for each gaze-orienting individual in the image, and computed their mean location as the model estimation. Note that the current model only takes the original image and just one individual’s head region as input rather than multiple head regions at the same time. This is a common limitation in current models and could cause the difference between machines and humans in their ability to integrate context information to perform the task. (Chong et al., 2020). We also tested other methods to integrate the model estimates across gazing individuals, including the median and the intersection gaze direction vectors pointing to the maximum probabilities in the prediction maps. The mean estimation resulted in higher accuracy.

To calculate human and model estimation error, we calculated the Euclidean pixel distance between the human or model estimation to the designated target person’s head location. Section 3.2-3.3 presents data based on target-absent images, while section 3.4 presents data based on both target-absent images and target-present images.

We assessed whether human estimations are more consistent with other humans than when compared to the model. Because most of the variance across images in the attended target location is in the horizontal direction (compared to the vertical direction; the ratio of variance = 18), we calculated the correlation of x-coordinate estimates between human pairs and between human and model pairings. We sampled 10,000 random human pairs and all possible pairings of humans and the model. The mean correlation between humans and the model $(r=0.50)$ was comparable to the mean correlation between pairs of humans $(r=0.55)$ (Appendix Figure 4).

We also evaluate whether humans tend to be more consistent in their overall estimations across images than the model. With 10,000 bootstrapped samples of human and model errors, we found a significantly smaller variance of human errors ($\sigma^{2}=1920$) compared to the model ($\sigma^{2}=6727$), indicating humans have more consistent estimations across different images.

First, we conducted one-way ANOVA to test the effect of the number of looking individuals presented in the image on both humans and the model estimation error. We found a significant main effect of the number looking individuals on human estimation error $F(2,5061)=19.3,p<0.001$, Tukey posthoc test showed a significantly larger error with only one looking individual compared to either images with two or three individuals (both $p<0.001$). In contrast, varying the number of looking individuals in the images did not significantly influence the model’s estimation error $F(2,196)=0.5,p=0.58$ (Figure 2). A 3 (number of gaze orienting people) $\times$ 2 (human vs. model) two-way ANOVA also confirmed a significant interaction between human and model $F(2,22187)=4.47,p=0.01$. This result indicates that humans are able to integrate multiple gaze cues (individuals) to improve the estimation accuracy, while the model does not benefit from the additional information.

We evaluated the humans’ and the model’s estimation error in instances in which the designated ”looked at” target was present vs. absent. Figure 3a and 3b show the co-registered locus of attention estimations relative to the location of the target person (origin $(0,0)$), for humans (3a) and model (3b). Two-way ANOVA: estimation source (human vs. model) $\times$ target presence condition (present vs. absent) showed a significant main effect of target presence condition $F(1,690)=134.9,p<0.001$, and a significant interaction between estimation source and target presence condition $F(1,690)=24.2,p<0.001$. A Tukey posthoc test showed that: 1. Both humans and the model had significantly lower errors when the target was present compared to when the target was absent (both $p<0.001$). 2. Humans had significantly higher errors compared to the model when the target was absent, $p<0.01$, but had significantly lower errors when the target was present, $p<0.01$. This indicated that the presence of the attended target in the image facilitates both humans and model to make more accurate judgments about the locus of attention, but with a much larger benefit for humans (Figure 3c).

Results also show a significantly higher amplitude of human estimation errors relative to the model in the vertical direction regardless of the presence of the attended target (Figure 3d, all $p<0.001$). For the errors in the horizontal direction, there was no significant difference between humans and the model when the target was absent (Figure 3e). However, when the attended target was present, humans errors in the horizontal direction were significantly smaller than the model, $p<0.001$, indicating a strong influence from the presence of the attended target that the model did not show.