Computational Attention System for Children, Adults and Elderly

In this section we present the evidences of age-impact on scene viewing behavior from developmental literature in order to establish the need for quantifying the age-impact on scene viewing to develop an age-adapted saliency model. There are many studies in psychology and neuroscience investigating the aging effect on eye-movement controls (Luna et al., 2008; Aring et al., 2007; Irving et al., 2006; Fukushima et al., 2006; Klein, 2001) such as focusing on target, maintaining focus on object, moving eyes effectively, coordinating both eyes and eye-hand coordination. These developmental aspects of eye-movement control have been studied mainly by using scene viewing behavior with the use of artificial stimuli. However, there are only a few studies employed natural stimuli to reveal developmental changes in free viewing behavior (Açık et al., 2010). Most of the developmental studies has either focused on developmental changes during early stage in life (childhood) or during late stage in life (elderly) (Aring et al., 2007; Ygge et al., 2005; Fioravanti et al., 1995; Luna et al., 2004), and their are only a few investigating the age-related changes for whole life span (children, adult, and elderly) (Irving et al., 2006; Açık et al., 2010) .

The studies reporting development of fixation system suggests that ability to fixate on target is present since early in childhood but the more complex aspect of fixation system such as stability and control of fixation increases with increasing age between 4 to 15 years of age (Aring et al., 2007; Ygge et al., 2005). Further, the studies about development of saccadic control system found that saccade velocity increases during childhood with peak value between 10-15 years of age group and then followed by decay until 86 years of age (Irving et al., 2006; Fioravanti et al., 1995). Findings about saccade latency i.e. reaction time to initiate an eye-movement have showed that the voluntary eye-movement decreases exponentially from birth to 14-15 years of age (Fischer et al., 1997; Fukushima et al., 2006; Klein, 2001; Luna et al., 2004). A study (Fischer et al., 1997) about anti-saccade (AS) task (which helps to understand the developmental changes in the cognitive ability to inhibit the reflexive saccade) in 300 participants of 8-65 years age has observed strong developmental effect in AS performance. The participants of age 40-65 years of age demonstrated a moderate deterioration in AS performance. These results are also replicated in several other studies where they found the relationship between age and anti-saccade is curvliner from childhood to adulthood (Klein and Feige, 2005; Luna et al., 2004) and from adulthood to elderly stage it has been found to be linear (Nelson et al., 2000; Fukushima et al., 2000; Klein and Feige, 2005). Even though this selection of the developmental literature briefly discuss the age-impact on eye-movement control but the neglected aspect of these works is that these findings are gathered by using artificial stimulus.

There are only a few studies employed the naturalsitic stimuls to understand the age-impact on scene viewing behavior (Açık et al., 2010). The study (Açık et al., 2010) found that bottom-up features of a scene such as color, luminance, contrast, etc. guided viewing during early stage in life (7-9 years) whereas in later stage (more than 72) dominated by more top-down processing. This result can be helpful in upgrading those saliency models based on guided search theory (Wolfe et al., 1989; Cave and Wolfe, 1990) to make them age-adapted, by tuning the bottom-up and top-down maps according to the oberver’s age-group, before combining them into a final saliency map. In this study, total number of fixation landed, inter-fixation distance, and performance in patch matching task has been used as other metrics to quantify the age-impact over scene viewing behavior.

The developmental literature presents several evidences that the age-related difference in natural scene viewing behavior is significant across children, adult, and elderly participants. Further, these findings motivates us to investigate the age-related differences in scene viewing with aim of reflecting them in development of an age-adapted computational model of saliency prediction. In order to develop and age-adapted saliency model it gets important to understand the age-impact on gaze distribution (fixation locations distribution) across different age groups, however, the studies reported in developmental literature has mainly focused on revealing the age-impact by analyzing the gaze attributes such as fixation duration, saccade amplitude, blinks, etc.

The computational models of saliency prediction which are reviewed in this section have in common that they are build on the psychophysical theories of visual attention system. Treisman’s Feature Integration Theory (FIT) (Treisman, 1993) and Wolfe’s Guided Search Model (Wolfe et al., 1989) has been among the most influential theories of psychological attention system. The main idea of the computational systems based on these theories is to compute different features maps in parallel and then fuse them together to get the final saliency map.

Most of the Bottom-up feature based saliency models have a very similar general structure to compute saliency. In these models we observe following basic structure: (a) Bottom-up features such as color, intensity, and orientation are extracted over multiple scales. (b) All these features are processed in parallel, to obtain the feature maps. (c) Finally, these features are integrated to obtain the final saliency map. Itti et al.’s model (Itti et al., 1998) is one of the most well-known model based on this general structure of saliency prediction. The Graph Based Visual Saliency Model (GBVS) (Harel et al., 2007) follows the similar general structure, where the maps are represented as fully connected graph and the distribution in Markov chain is treated as the saliency map.

Recent studies attempted to incorporate additional cues with bottom-up features such as human intention, given task, and cognitive states of mind which are also known as top-down features. Torralba et al. (Torralba, 2003; Torralba et al., 2006) proposed models for attention guidance which integrates the bottom-up features with the scene-context by using a Bayesian framework. Similarly, the SUN model of saliency prediction (Zhang et al., 2008) combines bottom-up feature maps with top-down information represented as Difference of Gaussian (DoG) and Independent Component Analysis (ICA) in order to predict the salient locations in imgaes. In very recent studies supervised learning based saliency models using eye-tracking data collected over young-adults are getting popular. Judd et al. (Judd et al., 2009) proposed a model which learn to predict saliency by learning feature (bottom-up and top-down) weights in a supervised manner over 1003 images viewed by 15 young adults. In light of the recent popularity of learning based saliency models there are many eye-gaze data introduced as shown in Table 1. It can be clearly seen form the Table 1 that participants of these experiments were mostly adults across all the dataset. Thus, all the computational models of visual attention learned on these dataset are inclined to reflect the scene exploration behavior of adults only.

The dataset used in this study was collected by Açiket al. (Açık et al., 2010) and the data can be retrieved from (Wilming et al., 2017). Fifty-eight participants participated in this study, comprising children (18 participants, age-range 7 to 9 years, mean age 7.6), adults (23 participants, age-range 19 to 27 years, mean age 22.1), and the elderly (17 participants, age-range 72 to 88 years, mean age 80.6) groups. All participatns reported normal or corrected-to-normal vision, including the elderly participants. Acik et al. (Açık et al., 2010) declared that all participants or their parents signed a written consent to participate in the experiment. The experiment was conducted in compliance with the Declaration of Helsinki as well as national and institutional guidelines for experiments with human participants.

In this study, we used 192 color images belonging to 3 different categories; “naturals”, “man-made” and “fractals” (64 in each categories). “Naturals” image category is representing natural scenes having trees, flowers, and bushes, this image category did not contain any artificial objects. “Man-made” category includes urban scenes such as street, road, building and construction sites, and the “fractals” are the computer generated shapes taken from the different web database such as Elena’s Fractal Gallery. Stimulus randomizations were balanced across pairs of participants. All images had a resolution of $1280\times 960$. EyeLink 1000 was used to record the gaze in its remote and hand-free mode.

Eye gaze was recorded while the observers viewed the images displayed for 5 seconds. The scene was subsequently replaced by a circular patch and participants had to determine if the patch was part of the previous scene or not. The patch recognition task was included to maintain the motivation of participants; thus the recognition results were not used in this study. Target stickers were placed on each observer’s head to compensate for head movements. Observers viewed the stimuli from a distance of 65 cm on a 20-inch LCD monitor display (width: 40 cm). All observers were instructed to explore the scene. Fixation and saccade were identified via a fixation detection algorithm supplied by EyeLink.

We generated human fixation maps, human saliency maps and heat maps from the gaze data. For each age group the human fixation map for an image stimuls was generated by combining the fixation landings of all observers of that age group. Further, the human saliency maps were generated from human fixation maps by convolving a Gaussian, similar to Velichkovsky et al. (Velichkovsky et al., 1996). The heat map was obtained from the human saliency maps to visualize the age differences in the region of interest.

The human visual system has the tendency of focusing earlier on the objects placed in the foreground than the objects in the background of the scene (Jansen et al., 2009). Gautier and Le Meur (Gautier and Le Meur, 2012) investigated the influence of disparity on saliency for 3D conditions. Their results indicate that foreground features play an important role in attention deployment. Itti and Koch (Itti and Koch, 2001) suggested that stereo disparity could be used as additional cue in saliency detection. However, this tendency of the human visual system was mainly explored for three-dimensional media but there is no study investigating the role of depth bias for two-dimensional media. Most importantly, to the best of our knowledge, for the first time we reported the role of scene depth in gaze landings across different age groups.

In the presented study, we visualized the age-related differences in gaze landings to the most salient foreground and the background regions in the scene. However, it is important to develop a framework to quantify this tendency across age groups. This tendency was observed in all those images in which some interesting item was placed in the foreground and the background of the scene, there was 20 such images in man-made category.

We propose a novel framework to quantify the depth bias tendency across age groups (Figure 3). As a first step a combined saliency map for each image is generated by linearly integrating the human saliency map of children, adults, and the elderly groups. The combined saliency map is generated to find the most salient locations independent of an observer’s age group. In the second step, we threshold the combined saliency map to get $t_{1}$ and $t_{2}$ percentage of the most salient locations of the scene. We fix $t_{1}$ and $t_{2}$ to five and ten, respectively, as we discovered that a small percentage of the most salient locations is sufficient to represent most of the fixation landings in the scene. In the third step, the most salient locations of the thresholded maps are manually labelled to foreground and background based on the near and far places in the scene. As shown in Figure 3, the thresholded map gives the most salient regions of the image (four different regions). Further, these regions are labelled as ”F” or ”B” according to their respective locations in the foreground or background of the scene (as shown in the ML map in Figure 3). Finally, the percentage of the fixations landing on the most salient foreground and background locations is calculated for each image to obtain an average score for individual age groups from the two thresholded maps. In summary, to measure the depth bias across age groups, we labelled most salient locations as foreground and background and then calculated average of the percentage of fixations landed on the foregrounds and backgrounds across all the images.

Explorativeness study is conducted to evaluate the age-related changes in eye-movement behavior across different image categories. We found that the observers belonging to different age-groups viewing the same set of images, have differences in explored regions, and this age-impact was present independent of the image categories used in the study. We used the explorativeness metric (Krishna et al., 2017) introduced previously to quantify gaze distribution of observers belonging to children, adults and elderly age groups. Explorativeness is quantified by measuring the first-order entropy of the human saliency map of an images, further, the entropy value defines the spread of fixations for different age groups. For an image, explorativeness is computed as follows:

where $U^{g}_{i}$ is the human saliency map of the $i^{th}$ image from all observers in group $g$ for which entropy is calculated and $h_{U^{g}_{i}}(l)$ is the histogram entry of intensity value $l$ in image $U^{g}_{i}$, and $L$ is the total number of pixels in $U^{g}_{i}$.

A higher entropy correspond to more exploratory viewing behavior of the observers belonging to an age group as their gaze points are more scattered, similarly a lower entropy correspond to less exploratory behavior of the observers.

Inter-individual similarity test was performed to understand the level of consistency between explored regions i.e. fixation locations among observers of the same or different age groups for all the image categories. Explorativeness analysis revealed that adults were more explorative than children and elderly participants; however, it is not clear whether the explored region by children and elderly observers was a subset of the explored region by adults (i.e. higher inter-individual similarity between children and adults age group). If that is the case, then adults can perform similarly in predicting the gaze landings of children observers to the children predicting others of the same age group. The inter-individual similarity aimed to answer this question in order to understand the need for development of an age-adapted computational attention system. We also aims to understand how the age impact differently to inter-individual similarity for the images belonging to ”nature”, ”man-made” and ”fractal” categories.

To evaluate the inter-individual similarity of an observer’s explored locations with rest of the observers of same and different age groups, we used source saliency map of one participant as a predictor to predict the target fixation of other observers (as in Figure 4). Thus under the intra-individual similarity the observer of source saliency map and target fixation points belongs to the same age group and for the inter-individual similarity the observer of source saliency map was different from target group. The formulation of this inter-individual similarity is as follows.

$A^{g_{i},g_{j}}_{k,n}$= The prediction score of the $n^{th}$ observer of group $g_{s}$ (source saliency map) in predicting others of a group $g_{t}$ (target fixations) for the $k^{th}$ image.

To measure the predictibilty of an oberver’s source saliency for target fixation points ($A^{g_{i},g_{j}}_{k,n}$) we made use of area under the curve (AUC) metrics as explained in (Le Meur and Baccino, 2013). The AUC-score measures how well the source saliency map of an observer could be used to find the pooled fixation locations of the rest of the observers of the same or different age group i.e. inter-individual similarity of the observer for an image with rest of observers. Final inter and intra-individual similarity score of $n^{th}$ observer is obtained by taking average over all the images ($K$) as shown in following equation.

Where $A^{g_{i},g_{j}}_{n}$ is inter-individual similarity of $n^{th}$ observer, $K$ is the total number of images.

Center bias is the tendency of fixating around the center locations in a scene while viewing the scene. This is one of the strongest bias reported in many eye tracking studies (Tatler, 2007; Zhang et al., 2008). The prediction accuracy of many computational models of visual attention (Zhang et al., 2008; Judd et al., 2009) have been seen to improve when center bias is included in their prediction framework. For example, a Gaussian blob centered at the middle of the image, considerably improved prediction performance of the learning-based saliency model (Judd et al., 2009).

To include the center bias tendency in the proposed age-adapted computational attention system, we first analyzed the age-related changes in this tendency. Then based on the analysis results we introduced a weighting factor for tuning the strength of the center-map in our proposed age-adapted model. To measure the center bias across age groups, we, first, generated a center-map for each age group by taking the average of all the maps for each age group. The heat map representation of the center-map over a sample image. Finally, the center bias for each age group is measured by measuring the euclidean distance between the centroid of the center-map and the center pixel of the image.

The results are shown in Figure 5, which show that:

• •

  Average of the fixations percentage landed on the most salient foreground locations in the images is significantly higher for child observers than for adults and the elderly (Figure 5(a)). The one-way ANOVA analysis suggest a significant influence of age over gaze landings in the foreground regions of the scene ($F(2,19)=2.99,p<0.05$), Further, a post-hoc analysis confirms that the foreground bias tendency in children is significantly different from that of adult and elderly observers ($p<0.05$).

  

  

• •

  The percentage of fixation landings on most salient background places (Figure 5(b)) gives an interesting observation for the gaze behavior of elderly observers. It was found that elderly observers have a significantly higher tendency for fixating over items placed in the scene background than children and adult observers. The one-way ANOVA analysis confirmed the age-impact over the background bias ($F(2,19)=3.56,p<0.03$), and post-hoc analysis revealed the behavior of elderly observers is significantly different to adults ($p<0.03$) and children ($p<0.03$).

The explorativeness result suggests following tendency for children, adults, and elderly observers:

• •

  Result showed significant impact of age over explorativeness tendency across all the image categories ($F(2,191)=179.45,p<0.001$). Post-hoc results across all the image categories showed that the child observers possess the lowest exploratory tendency compared to adults ($p<0.001$), who are highly explorative for the same set of images. Elderly observers showed less exploratory tendency than adults but higher than that of children ($p<0.001$) (Figure 6).

• •

  Explorativeness tendency reflects the level of details an observer visualizes in a scene. With the highest exploratory behavior, young adults have a tendency to visualize finer to coarser details in the scene, as inferred from (Judd et al., 2011), which states that a decrease in spatial scale results in a decrease in the exploratory behavior of the observers.

The age-related differences in explorativeness tendency on three different class of images (see Figure 6) show that this exploratory behavior is independent of the type of stimuli being observed.

The inter and intra-individual similarity results are shown in Table 1 and Figure 7.

• •

  One-way ANOVA confirmed the age influence over intra-age-group agreement between observers independent of the class of the image ($F(2,192)=41.39,\newline p<0.001$). Further, post-hoc revealed that the children are in better agreement with each other for explored locations than adults and elderly observers for all the image categories, $p<0.001$ (Figure 7(a) and Table 1).

• •

  Inter-age-group agreement analysis suggests that fixations of an specific age group can be best predicted by an observer of the same age group. As highlighted in table 1, the diagonal entries in each image class are highest compared to other elements in the same row, which suggest a fixation is well predicted by an observer of the same age-group than the observer of different age-group. For instance, as shown in Figure 7(c) for natural images, a child can predict well the fixations of other children better than an adult can predict children observers ($F(2,63)=115.95,p<0.001$).

This result suggest that the prediction accuracy of learning-based saliency models can be optimized only when the training weights are learned over the age-specific fixation data.

We can visualize different tendency of gaze landings towards center of scene for different age groups as shown in center map of different age groups (Figure 8). Agreement score (AUC score) of fixations of different age groups towards their center map revealed the age-impact over this tendency independent of the image categories $F(2,63)=34.10,p<0.001$ (see Figure 9). The post-hoc analysis revealed that children have significantly different tendency towards center map that adults for all the scene categories ($p<0.001$).

Further, the result of Euclidean distance from the centriod of the explored region by different age group and center pixel of the scene showed that children have highest center bias but elderly participants showed similar tendency. (Nature- 150, 210, and 184 are the euclidean distance in pixels for children, adult and elderly participants, man-made- 203, 267, and 244, and fractals- 189, 214, and 202).

The analysis results recommended five major findings that we use to develop an age-adapted computational model of visual attention. These recommendations are as follows: .

• •

  Depth object bias tendency showed that child observers are focused more towards the objects in foreground compared to adult and old observers. This can be incorporated in age-adapted model by including a depth map at different spatial scales for children and old age groups.

• •

  Explorativeness result showed that children were least explorative among the three age groups i.e. they are mostly focused on the coarser details of the scene, whereas being highly explorative adults evaluated finer detail too. This can be included in age-adapted model by selecting different spatial scale for different age groups.

• •

  Inter age group agreement analysis suggests that the saliency maps of adults fail to appropriately predict fixation landings of child and old observers. Thus it is advisable to train the learning based models over the age-specific gaze data to optimize the prediction performance for different age groups.

• •

  Analysis result of age-related changes in central bias tendency motivated us to investigate the scope of reflecting these change in computational model of visual attention by tuning the center feature map of existing algorithms.

We use a learning approach to train a classifier directly from age-adapted eye tracking data. The general structure is that the proposed model learns the weight for combining different features extracted from a input image to generate the final saliency map.

Features that we used for training and testing can be categorized in low-level, mid-level, and high-level features. Low-level feature includes, color features, itti’s saliency map (intensity, color, and orientation channel) (Itti et al., 1998), Torralba’s saliency map (Torralba, 2003), and the local energy of steerable pyramid (Simoncelli and Freeman, 1995). In mid-level features horizon feature is extracted similar to (Judd et al., 2009), as the human’s have tendency of naturally directing their gaze on horizon locations in a scene viewing. For high-level features, object and car detection algorithms were applied in the proposed model. Figure 12 shows the feature maps extracted from a sample image.

The center bias is included by a adding center map generated by Gaussian blob placed at center of the scene as in (Judd et al., 2009). The most important contribution of this work is that for the first time we included a depth feature maps extracted from input 2D scenes in context of the age-related diffrences in depth bias tendecies. The depth maps from single images are estimated using a method proposed in (Liu et al., 2015) as shown in Figure 11.

We used Liblinear support vector machine to train our model over different features extracted from the scene. The model he model of Judd et al. (Judd et al., 2009) was not well suited for the age-adapted mechanism as the depth bias features maps were not included in the proposed model. To fit the training and testing in our age-adapted framework, we follow the recommendations of our analysis results. We can see in Figure 12 different features are extracted at several spatial scales (finer to coarser), where band 2 to 4 is scale 1 (finer), band 5 to 8 is scale 2 and band 9 to 12 is scale 3 (coarser). Further, based on the results of explorativeness study, first we focus on feature scale selection, where we found that being less explorative only a few scales are required to represent the level of details explored by children and old observers, conversely, for adults, all the scales are used to represent the level of details they explored in a scene. From our experiments we found that optimal scale for children was 3, whereas scale 2 and 3 for old and scale 1, 2, and 3 for adult observers.

We split the dataset with 192 images into 120 training and 72 test image set. Further, different features were extracted at all the scales, and only a few scales were chosen to represent the age-adapted changes in scene viewing behavior. The use of center map feature was altered by including a weight factor for tuning the age-related differences in center bias tendency as reported in our analysis. Top 10 strong positive and negative samples from top 5% of the human saliency map and similarly 10% strong negative pixels from bottom 20

Further, as suggested by the agreement score analysis, strong positive and negative samples from the age-specific human saliency maps were used for training as it was seen that saliency maps of adults were poor predictors for the fixation landings of children and old observers and it is advisable to use the saliency maps of the child observers and old observers to predict fixation landings of observers of these individual groups respectively.

For a given set of feature and levels (positive and negative samples) selected over the age-based human saliency map, SVM is used to learn the optimal weights for depth features i.e. model parameters for each age group in order to predict salient locations for different age groups. Further, for a given image of test dataset, a final saliency map is generated by combining different feature extracted at different scales over an age-specific optimal weights learned for newly included depth map during training the model. Although the prediction accuracy is mentioned for different image categories as shown in Table 2 but the model parameters are learned over all the images independent of the class of the image.

Where $w_{g}$ and $b_{g}$ are optimal weights for age group $g$, i.e., children, adults, and old separately, $F(I_{i})$ is feature vector for the $i^{th}$ test image, this vector is of different size for different age group due to the proposed optimal feature scale mechanism. Based on the saliency values we classified the local pixel as salient or not.

Performance of our proposed age-adapted model was evaluated using AUC metrics (Judd et al., 2009). The evaluation score is listed in Table 2 and the resulted saliency map for sample images are shown in Figure 13. We compared our model with several state-of-the art methods reported in (Itti et al., 1998; Harel et al., 2007; Jansen et al., 2009). The result shows that the proposed age-adapted model outperform the existing saliency models for all age groups independent of the image categories. It is also interesting to note that children observers show better prediction accuracy independent of the saliency model, however the age-adapted model optimized the prediction accuracy each age groups, which suggest that our modifications especially including depth map in leaning based model works well for adult observes as well.