Predicting Visual Attention and Distraction During Visual Search Using Convolutional Neural Networks
^*This is a pre-print manuscript currently under review in Journal of Vision.

In our modeling, we employ the light-weight convolutional encoder-decoder architecture of MSI-Net (Multi-scale Information Network) free-viewing saliency model [3]. Our network has an additional encoder stream that receives a sample target image from the search target category. The sample target fed into the target stream is not the same as the target object in the search image. At each epoch, we randomly choose one sample target image from the five samples that we have for each of the 18 target categories and feed it to the target encoder stream. These sample target objects can be seen in figure 24.

Thus, one encoder stream of our network extracts features from the search image and the second encoder stream extracts features from a sample image of the target category. These two streams are identical and their parameters are shared. Each encoder stream is composed of two parts. The first part contains a VGG16 convolutional neural network [34] pretrained on ImageNet dataset [35], and the second part is an ‘ASPP’ [36] module. VGG16 is composed of 5 blocks of convolutional layers with 3x3 kernels followed by pooling layers, and three final fully-connected layers. For the purpose of feature extraction, we remove the last fully-connected layers. All layers use zero-padding to keep the width and height of their input tensors unchanged. Hence, only pooling layers cause reduction in dimensions. As in MSI-Net, we remove the strides of the last two pooling layers in VGG16 to keep the width and height of the features as $1/8$ of the input image. To be able to use the pretrained weights on Imagenet and match the same number of parameters, we use a dilation rate of 2 in the last three convolutional layers. To obtain a multi-scale feature representation, we concatenate the output of the three last pooling layers, all of which have an output dimension of $1/8$ of the input image. The pooling layer in block three has 256 output channels and pooling layers of blocks four and five each has 512 channels. Hence the concatenated features tensor has 1280 output channels. This tensor is then sent to the second parts of the feature extraction block called ‘Atrous Spatial Pyramid Pooling’ abbreviated as ‘ASPP’.

ASPP is a semantic segmentation module that is composed of convolutional layers with different dilation rates (‘Atrous’ convolution is another term for dilated convolution). This causes the layers to use filters with different effective fields of view, that capture objects and contextual information at different scales. Same as MSI-Net, the ASPP architecture used in our network is composed of six convolutional layers. Four convolutional layers receive the extracted feature tensor (with 1280 channels) directly as their input. One of them has 1 × 1 kernels and only perform point-wise non-linearity without learning spatial relationships. The other three have 3 × 3 kernels with 4, 8, or 12 dilation rate, which allows extracting multi-scale dependencies from the feature tensor. The fifth convolution layer receives the mean of the features across spatial dimensions, which is believed to contain global contextual information of the image. This convolution then applies point-wise non-linearity with 1 × 1 kernels to the mean tensor. The output of this convolution is then upsampled to match the spatial dimension of the original feature tensor. Finally, the output of all these five convolutions are concatenated to form a single tensor. As the output of each convolution has 256 channels, the concatenated tensor yields 1280 channels. The concatenated tensor is then passed through another convolutional layer with point-wise 1 × 1 kernels and ReLU non-linearity. The output of this layer yields the final multi-scale feature tensor with 256 channels. The architecture of ASPP can be seen in image 1.

Both the target and stimulus streams use the same encoder and ASPP architecture to extract features from the sample target image and search image. Then extracted features from the target and search image are convolved together. This convolution is performed using a convolutional layer with the target feature tensor treated as the filter and the search image feature tensor as the input to the layer, without any non-linear activation function. The output of this convolution has 256 channels, which is then passed through a decoder to retrieve the spatial dimension of the input image. Decoder is composed of three bi-linear up-sampling blocks, followed by 3 × 3 convolution layers to avoid checkerboard artifacts. There are four convolution layers in the decoder with 128, 64, 32, and 1 filters. The first three convolutions that come immediately after up-sampling layers are followed by ReLU non-linearity; while, the last convolution layer that generates the output FDM is not modified by a ReLU. The output values are then normalized between 0 and 1, such that all values sum to one. This modification converts the generated map to a probabilistic density map. A visualization of our network is presented in figure 2.

We created the fixation density maps for each image by blurring each fixation point with a Gaussian kernel centered at that point with a standard deviation of 11. We sum these Gaussians on the image plane, and then normalize the map by dividing it by the sum of pixel values to convert the density map to a probability distribution. Figure 3 shows a sample fixation density along with its heatmap. The original images of COCO-Search18 dataset are 1050 by 1680 pixels. For more efficient computation we resize all search images to 320 by 512 pixels.

We ignored the first fixation point in each trial, as all observers were fixating on a dot in the center of the screen at the onset of displaying each search image. We also removed the trials with incorrect responses from our data, i.e. when observers reported the wrong object as their detected target. We use the same train/valid split as was used in [22] in our modeling. After removing the incorrect trials, there remained 2150 training and 324 validation task-image pairs in COCOSearch18 dataset. Some of the search images are used more than once for different search targets. Hence we defined task-image pairs to distinguish between the search targets in similar search images. From 2150 training task-image pairs, we then separate 324 pairs for the test dataset, such that there are 18 task-images from 18 different categories. Hence, our train-validation-test split contains 1826-324-324 task-image pairs. We further augment the training data by horizontally flipping the search images along with their ground-truth fixation density maps. The reason we chose horizontal flipping was inspired by [37], which suggests that among common transformations such as cropping, flipping (mirroring), rotating, changing brightness, and shearing, horizontal flip causes the least change in the fixation maps. After the augmentation, the train-validation-test split becomes 3652-324-324 task-image pairs. A histogram of the number of task-image instances for train, validation, and test splits over target categories can be seen in figure 4.

We initialize the VGG16 network with its pre-trained weights on ImageNet dataset for object recognition task, and then fine-tune (by adjusting all layers’ weights) it on COCOSearch18 for our problem. It has been shown that high-level features extracted from object recognition networks can improve the performance of free-viewing saliency prediction. We therefore apply the same technique for visual search saliency. The weights of VGG networks used in both streams are shared. This weight sharing reduces the network’s complexity and training time significantly, while also improving the accuracy.

We train the network on COCOSearch18 dataset by minimizing KL-divergence loss in equation 1 between the predicted and ground-truth fixation density maps. We use the Adam optimizer with learning rate of $10^{-5}$, and train the network for 8 epochs, with batch size of 1. The input search images are all resized to 320 × 512, and the target images are resized to 64 × 64. The output fixation density map dimension is also 320 × 512 same as the input search image.

We evaluate our model using 8 saliency metrics: area under ROC curve (AUC Judd), AUC Borji, shuffled AUC (sAUC), normalized scanpath saliency (NSS), Kullback–Leibler divergence (KLD), Pearson’s correlation coefficient (CC), similarity measure (SIM), and information gain (IG). Among these metrics, NSS, IG, AUC Judd, AUC Borji, and sAUC are considered location-based, while KLD and CC are distribution-based [38]. For location-based metrics, groundtruth binary fixation maps are used, and for distribution-based metrics, groundtruth blurred fixation maps (convolved with Gaussian kernel densities) are used in evaluation. To compute KLD, the predicted and groudtruth blurred fixation maps are first converted to probability distributions, using a division by the sum of pixel values. Then formula 1 is used to compute the dissimilarity between the groundtruth and prediction. IG needs the predicted and baseline saliency maps to be normalized between 0 and 1 (using min-max normalization) and then divided by their sum to be converted to a probability distribution. Baseline saliency map contains all blurred fixations from all other images (overlayed and normalized). Then for all fixation locations in the groundtruth binary fixation map, we calculate the information gain between the predicted and baseline saliency map. To compute SIM we also normalize the maps between 0 and 1 (to avoid a erroneous performance boost when a model assigns a nonzero value to every pixel) and divide them by their sum in order to ensure that the maximum value of score will be 1. Then we compute histogram intersection between the groundtruth and prediction. In NSS computation, the predicted maps are normalized such that their mean is zero and their standard deviation is 1. Then at the fixation locations in the groundtruth binary fixation maps, we compute the mean value of the normalized predicted saliency map. Similarly, in computing CC we normalize the prediction and groundtruth blurred fixation maps to have zero mean and standard deviation of 1, then we compute the correlation (dependence) between the groundtruth and prediction using formula 2.

AUC (Judd), AUC Borji, and Shuffled AUC all first normalize the predicted maps between 0 and 1 (using min-max normalization). AUC Judd computes the area under the ROC curve created by sweeping through threshold values determined by the range of saliency map values at fixation locations. AUC Borji and sAUC compute the area under ROC curve created by sweeping through threshold values at fixed step size until the maximum saliency map value is reached. Another difference between these three variations of AUC comes from the definition of false positive rate. AUC (Judd) computes the ratio of the non-fixated image pixels that have values above threshold in the predicted saliency maps. AUC Borji computes the the same ratio but instead of computing at all the non-fixated image pixels, it generates N splits (by default 100) of random locations sampled uniformly from all image pixels (the number of random locations in each split is the same as the number of fixation locations), then computes the ratio for each split and averages the ratios over the N split. Shuffled AUC computes false positive ratio similar to AUC-Borji, but instead of randomly choosing from all image pixels, it uniformly samples from binary fixation maps of other images (we use only one other fixation map in our implementation) to create the splits and compute the average ratio.

As in [3], we also evaluated our predicted fixation distributions without applying any metric-specific post-processing methods. In order to obtain an accurate and loss-less evaluation, we store the predicted and groundtruth fixation maps as numpy files instead of images. The results of the evaluation on unseen test set extracted from COCOSearch18 dataset is summarized in table IV. The overall performance of our 2-stream model is obtained on all target categories on a test set with 324 images, containing 18 sample images for each of the 18 target categories. We further compare the performance of our unified 2-stream model with single category-specific networks trained separately for each target category. Our current unified model has two streams that allow it to receive information about the target category, and output target-specific FDM predictions based on the target. An alternative approach is to train separate one-stream networks for each of the 18 target categories. As these network are inherently target-specific, they do not need a separate stream for a sample target. For a better comparison, we report the results of our two-stream network on test images of separate categories (18 test images for each category) along with the results of single category-specific trained models in table I, II, and III. Although in some cases, one-stream models trained on one category achieve slightly better performance, a unified two-stream model that works for all categories demands less memory, is easier to train, and is more biologically plausible. Also, as the weights between the target and image streams are shared, a two stream model has the same level of complexity and number of parameters as the one-stream network. A slight increase in the training time of the 2-stream model is caused by a bigger training and validation set at each epoch, as we train on all images rather than only images of a single category. We also took the average performance of our category-specific 1-stream models over 18 target categories, and report the results in the second row of table IV. To clarify the impact of a separate target stream in our unified model, we report the results of a one-stream model trained and tested on all categories, in the third row of table IV. By comparing the first and third rows in table IV, we see a boost in performance of the model across all saliency metrics especially NSS, SIM, and IG.

To gain insight into the qualitative performance of our model, we visualized the network’s predicted density maps and compared them to the ground-truth density maps for the test dataset. Some of these visualizations are shown in figures 5 and 6. The qualitative results show that the network accomplishes to detect the targets in most of the images. It also learns where to look for each target category based on its experience in the training data. For instance, when looking for oven or toilet, the locations closer to the ground are considered salient; however, for bottle, bowl, laptop, fork, knife, mouse, and cup, the top of surfaces such as tables are mostly salient. While searching for TV and clock the higher items in images are often considered salient. Moreover, the items sharing some features with the objects of target category are considered salient both by the network and human observers. As an example, when observers are looking for a bottle, objects such as glasses and cups that are also relatively cylindrical, are fixated. In general, the network’s predictions and ground-truth FDMs have great similarities with matching distraction patterns.

We also tested our network on the images captured from Couche-Tard laboratory store at McGill campus. Some of these images are shown in figure 7. We do not have the ground truth eye fixation data on these images to compare with our predictions; however, it can be seen that the network generates qualitatively reasonable results, especially for images which possess the same level of clutter as COCOSearch18 dataset. In figures 7-a and 7-d the level of clutter is very high and all of the objects also belong to the ‘bottle’ target category. This causes the network to predict a large salient region for these images. Also, our network is mostly trained on images that only contain one instance of the target category, thus it often fails to detect all targets as salient when more than one target object is present in the image. We can see this behavior in figures 7-c and 7-b, where only one instance of target is selected as salient. In figure 7-c, the network makes a reasonable prediction that the observer will be distracted by the discount offer on coca-colas. In figure 7-f, the network predicts that when the observers are searching for the cup they most likely get distracted by the circular plastic caps on the side of the espresso machine. In figure 7-g, the network predicts most of the objects which have the same shape as a microwave, have steel cover, and are located higher in the image, as distracting.

As an ablation study, we investigate the effect of data augmentation, presence of ASPP, presence of decoder, initializing VGG with random weights, and not sharing weights between the target and search image streams.

We realized that with data augmentation, using horizontal flipping, the performance of the model increased on most saliency metrics, as can be seen in table V. The average AUC-Borji, and sAUC are slightly lower for augmented model, but considering the standard deviation of the results they seem to be around the same range. Adding more augmentations such as vertical flipping did not lead to performance increase. The reason might be due to the effect of vertical flipping transformation on the fixation pattern. The fixation map of a vertically flipped image might no longer be in accordance with the vertically flipped fixation map of the original image. This would create erroneous data, which causes the model’s performance to drop.

To explore the effect of ASPP architecture, we removed it and directly fed the concatenated VGG16 features to a 1 × 1 convolution with 256 filters. Then we passed the output through the decoder. As can be seen in table VI, the results validate the effectiveness of ASPP architecture in feature extraction.

As a second architecture change, we removed the decoder from the network, and up-sampled the output of the convolved streams by 8 in one shot. Then we normalized the output to create a probabilistic density map. Removing the decoder leads to checkerboard artifacts in the predicted density maps, and the maps seem to be more sparse.

We realized that initializing the weights of the feature extractor backbone network, i.e. VGG16, randomly instead of using pretrained weights on ImageNet, dramatically decreases the performance.

We performed vast experiments on the size of the input sample target image, and chose 64x64 dimension as the optimal resolution.

We further investigate whether pre-training the one-stream category-specific models on SALICON free-viewing dataset can enhance their performance on COCOSearch18. Unlike the free-viewing scenario where pre-training on SALICON dataset leads to performance boost, in case of visual search saliency, pretraining on SALICON degrades the performance. The main reason comes from the significant difference between a visual search FDM and free-viewing FDM. In the free-viewing condition, the gaze locations are more spread out over the image, while in visual search gaze is more focused on the target object. Also SALICON has 10000 training samples, almost three times larger than our augmented dataset. Hence, retraining the network on COCOSearch18 after it is pre-trained on SALICON does not change its behavior significantly, and the predicted FDMs are too scattered.

To understand how our network predictions for visual search differ from a one-stream MSI-Net that is trained for free-viewing task, we tested free-viewing MSI-Net, trained separately on Salicon, MIT1003, and Pascals datasets, on COCOSearch18 dataset. The results are shown in figures 8 and 9. In general, a vanilla MSI-Net trained on free-viewing datasets predicts a larger part of images as salient compared to our double-stream visual search model. Sometimes, the distractor objects that our network predicts as salient is also considered salient under free-viewing condition; however, this is not always the case. Also, in freeviewing saliency, the target objects are not always parts of the salient regions.

Same as MSI-Net, our model is relatively light-weight. As the weights are shared between the two streams of our network, our model has 24,934,209 parameters similar to the one-stream MSI-Net. MSI-Net is among the lightest free-viewing saliency models, and here we exploit this feature by extending MSI-Net to a visual search saliency predictor.

To the best of our knowledge, the model we proposed here is the first attempt to predict distraction through estimating fixation density maps for visual search using COCOSearch18 dataset. Although the creators of COCOSearch18 have partially worked on fixation density maps in [22] and [23], their main concern is predicting sequential scan-paths.

One failure of our proposed model is its confusion in identifying similar target categories such as fork and knife. In most images where a fork and knife are both present, the network mistakes one target with the other. Also our network sometimes generates high fixation probabilities for non-target objects that share close similarity (such as similar color or shape) with the target. Hence the FDM peaks will be mistakenly placed on these non-target objects. This behavior has been also seen in the ground-truth data of human observers. Many of the fixated non-target objects by human observers have resemblances to the target. In most cases, humans are able to find the target after fixating on the similar non-target objects; while the network sometimes fails to do that.

One direction for future research could be on the choice of the sample target images. Convolutional neural networks are not inherently rotation-invariant. Thus, the difference between the orientation of the sample target image and the target object in the search image could affect the modeling performance. One possible solution might be to use a rotation-invariant convolutional neural network such as RIFD-CNN [39] in the target stream to extract rotation invariant features from the sample target images.

An application of our model could be in object detection. Predicting salient regions of a scene while searching for an object category can be used as an object proposal strategy in object detection algorithms. Salient regions highlight potential locations that might contain the target category in an image.

We use Mask-RCNN [40] instance segmentation network in our modeling. Mask-RCNN belongs to the RCNN (Regions with CNN features) group of networks. The first RCNNs, namely RCNN [41], fast-RCNN [42], and faster RCNN [43], were solely for object detection purposes. Mask-RCNN on the other hand provides pixel-level segmentation, by adding a new branch to faster-RCNN along with some modifications. We first describe faster-RCNN architecture and then extend that to Mask-RCNN.

As the first step, faster-RCNN extracts multi-scale image features using FPN (Feature Pyramid Networks) [44] architecture. FPN consists of a bottom-up pathway, and a top-down architecture with lateral connections. The bottom-up pathway consists of popular CNN architectures such as ResNet or VGG, which extracts features from input images. In our modeling, we use ResNet-101 as our feature extraction backbone. Top-down pathway generates feature pyramid map, which has the same size as the bottom-up pathway. To create the pyramid, as we go down the top-down path, we upsample the previous layer by 2 using nearest neighbors upsampling. We apply a 1 × 1 convolution to the corresponding feature maps in the bottom-up pathway, and add it to the upsampled previous layer element-wise. A 3 × 3 convolution is then applied to all merged layers, which reduces the aliasing effect when merged with the upsampled layer. FPN outperforms single CNNs due to creating high-level semantic feature maps at various scales.

As the next step, the network proposes several boxes in the image and checks if any of them corresponds to an object. For this purpose, faster-RCNN has a fully convolutional network, known as region proposal network (RPN), on top of the extracted CNN features. This network passes a sliding window over the CNN feature map; at each window outputting k potential bounding boxes and the scores of how good each bounding box is. The k proposed bounding boxes are compared to k reference boxes, which are called anchors. Each anchor is centered at the sliding window in question, and has a scale and aspect ratio. At each sliding position, there are typically 9 anchors with 3 different scales and 3 aspect ratios. To generate features for each of these proposed bounding boxes, the pre-computed CNN features of the whole image are reused through a method called ROI pooling. Then for each bounding box, we pass its features into faster RCNN to generate a classification and run a linear regression to fit a tighter bounding box to the object.

To generate pixel-wise segmentation along with object classification and bounding box detection, authors introduced Mask-RCNN. In mask-RCNN, a fully convolutional neural network is added to the top of CNN features to generate a mask segmentation output. This is in parallel to the classification and bounding box regressor network of the faster RCNN. Also, authors realized that the regions of the feature map selected by ROI pooling were slightly misaligned from the regions of the original image. Since the image segmentation required pixel-level specificity unlike bounding boxes these misalignments led to inaccuracies. The authors solved this problem by introducing a method called ROI Align, which forces the cell boundaries of the target feature map to realign with the boundary of the input feature maps.

COCOSearch-18 dataset does not include the segmentation of objects. Although all images in COCOSearch-18 are from the COCO dataset, which provides partial segmentation of objects, most of these segmentations contain neither the target nor the fixated distractors. Hence, we use COCO Annotator [45], an online annotation platform, to manually segment the fixated objects. Due to time constraint, we have annotated 3 search categories: bottle, bowl, and car, out of 18 available categories. The output of COCO Annotator is a json file similar to the format of COCO dataset. For more information on the content of the json files please refer to [46].

To understand what percentage of distractors are highly distracting, we assign a distraction level to each distractor. We define distraction level of a distractor, as the number of participants who have fixated on it. As we have 10 participants in COCOSearch-18, the distraction levels vary between 0 and 10. A score of 0 means no distraction (no fixation), and 10 means a high distraction. Figure 10 shows the histogram of distraction levels for fixated distractors in three target categories. As expected, most distractors are fixated by 1-3 observers. There are few cases where all 10 observers fixated at the same distractor. Figure 11 shows a segmented image mapped to RGB color space based on the distraction levels.

Using our annotation labels, we create the ground truth instance segmentation of targets and fixated distractors such that each object forms a separate channel in a one-hot encoded format, as in figure 12. In this figure, the first channel contains the target object and the other channels contain the distractors (the number of distractors vary for each image, but they are limited to 20 instances). Considering the complexity of Mask-RCNN and for a more efficient use of data, we do not divide dataset into train-validation-test sets. Instead, we combine the training and validation splits of COCOSearch-18 and use a 5-fold cross validation method to train and evaluate the model on the whole data.

Our segmentation problem is composed of 2 object classes ‘target’ and ‘distractor’ along with the default background class. The segmentation masks shown in figure 12 are then reconfigured in MASK-RCNN pipeline based on their segmentation class, as shown in figure 13.

We use ResNet101 with strides 4, 8, 16, 32, and 64, as the backbone feature extractor. We initialize MASK-RCNN with pre-trained weights on MS COCO dataset. Then we perform a two-stage fine-tuning with progressively lower initial learning rates. The momentum and weight decay are set to 0.9, and 0.0001 for both stages. In the first stage, we freeze all layers, except the head branches. The head branches are composed of RPN, classifier and the mask heads of the network. We train these head layers for 10 epochs with a learning rate of 0.002. In the second stage, we fine-tune all layers with a learning rate of 0.001. In both stages, we also augment the data with horizontal or vertical flipping, each being chosen with 50% probability.

We use a batch size of 1 for our training and validation. The network is able to detect maximum 100 object instances. The input images are of 320 × 512 dimension, and are transformed to square shape of 512 × 512 dimension. There are 5 different losses that are jointly minimized during training, in an end-to-end fashion. These losses are: RPN classification loss, RPN bounding box regression loss, M-RCNN classification loss, M-RCNN bounding box regression loss, and M-RCNN instance segmentation loss. All of these losses have equal weights (importance) and are calculated for both training and validation sets, at each epoch. During training the model tries to minimize the summation of all these losses, as seen in equation LABEL:eq:loss.

At the end of each epoch, we calculate mean average precision (MAP), mean average recall (MAR), and f1-score at IoU threshold of 0.5 for the validation set. We save the weights of the model if the f1-score of validation set has increased compared to the previous epoch. To compute MAP, we first compute precision and recall values for each image. Precision or positive predictive value indicates what ratio of the predicted objects are true positives, i.e. they correspond to actual ground truth objects (A true positive happens when the predicted object class is correct and the predicted mask has an overlap of more than IoU threshold of 0.5 with the ground truth mask). Precision formula is shown in equation 4. Recall or sensitivity, as shown in equation 5, indicates what ratio of all the ground truth objects are correctly predicted by the model.

Average precision (AP) can be calculated for each object class, as the area under the precision-recall curve for that class. Then mean average precision (MAP), is the mean of AP over all object classes (target and distractor). For mean average recall (MAR), we compute recall values for IoU threshold of 0.5, and take the average over images and object classes. When our M-RCNN has a high recall but low precision, it predicts most of the targets and distractors correctly but it also has many false positives and predicts many background objects as distractors (or sometimes targets). While, when it has high precision but low recall, then most of the objects it detects as distractors or target are accurate, but it misses some of the distractor and target objects. F1-score, which is the harmonic mean of precision and recall, finds a balance between these two metrics. F1-score is calculated using equation 6. A high f1-score indicates a high precision and recall.

As we need to train a separate Mask-RCNN for each target category and the amount of data for each category is comparatively small, we do not separate a test set from our data. Instead, we run 5-fold cross-validation, dividing data into 5 folds, using one different fold as the validation data at each round, and training the model on the rest. In each cross validation round, after the training ends, we compute the MAP and MAR of the best check-pointed model on the validation data of that round. After 5 rounds of cross-validation, we then compute the average MAP, MAR, and f1-score over all 5 validation folds. We repeat this cross-validation for 5 independent runs (25 validation rounds in total) and report the mean and standard deviations of the results in table VII. This method helps us to validate the model’s performance on all data and average out the variations in the difficulty of images among different validation sets. We also consider a baseline model, that is a MASK-RCNN initialized with pre-trained weights on MS COCO dataset without any further training. We compare our trained models for each of the three target categories with this baseline model in table VII. The Baseline model has a low performance with an average MAP₅₀ of 0.005, and MAR₅₀ of 0.087 and f1-score of 0.010, over the three target categories. This shows that our training improves the model performance considerably. Our trained model has an average MAP₅₀ of 0.570 , $114\times$ better than the baseline, MAR₅₀ of 0.726, $8\times$ better than the baseline, and f1-score of 0.637, $6\times$ better than the baseline. Here, the baseline model should be better than chance (a random model), as it is pre-trained on MS COCO dataset to detect objects. Yet, it has much worse scores compared to our trained model. This shows that our model performs much better than chance in predicting targets and distractors.

In figure 14, we show sample precision-recall curves for each of the three models of bottle, bowl, and car target categories. Each curve contains the precision and recall values on the validation data of a full run of 5-fold cross validation over the whole dataset. Clearly, the highest area under the PR curve belongs to car, then bowl, and finally bottle target category. This is in accordance with the MAP₅₀ scores we obtained for these categories, i.e. car has the largest and bottle has the lowest value.

In figure 15, we show the confusion matrices of each target category averaged over 25 runs (5 independent runs of 5 fold-cross validation). A prediction is considered true positive if the predicted and ground truth bounding boxes have an IoU of more than 0.5, and belong to the same object class (target/distractor). The ground truth objects that do not have an IoU of more than 0.5 with any of the predicted objects are predicted as background, and the predicted objects that do not pass the overlap threshold with any ground truth object has background label as their ground-truth. The accuracy of each class (target and distractor), which is computed using equation 7, can be seen in the last row of the matrices. The mean accuracy percentage and its standard deviation are shown in green font. We report these accuracies in table VIII.

The highest target detection accuracy belongs to car category, which can be related to the higher visibility and bigger size of target objects, and bowl category has the highest distractor detection accuracy. It can be seen in confusion matrices, that a significant number of background objects, $\approx 40\%$, are predicted as distractors by the network. We hypothesise that if more observers were involved in the fixation data of COCO-Search18 dataset (which are currently only 10 people), we would expect that some of the other background objects appear as distractors. It is possible that by increasing the number of observers, the percentage of false positive for distractor class decrease. Also, $\approx 16\%$ of distractors are not predicted by the model, we will see later that these are mostly the less distracting objects.

Figures 16, 17, 18, 19, 20 and 21, show sample predictions of M-RCNN along with their ground-truth segmentations. The network sometimes fails to detect the target mostly due to occlusion, blurriness, or a very small size. The network also outputs a confidence score/probability for each segmented object. In the generation of figures 16 to 21, we have defined various thresholds on the distractor’s confidence score to only output the top 3 or 4 predicted distractors in each image. Without defining these thresholds, the network shows all of the predicted distractors even those with low confidence scores, that do not even contain a real object. For most images a confidence threshold of  90-95% removes most false predictions. Our network segments and classifies many foreground objects that do not belong to the target category as distractors. However, factors such as similarity to the target, proximity to the target, proximity to the center of the image, and distractor’s size, affects how distracting an object is. Defining a threshold on the confidence scores could be a way to separate the most distracting objects.

We discussed earlier that we labeled each segmented non-target object with its distraction level, i.e. the number of observers who fixated on that object. To test if a M-RCNN can distinguish distractors with different distraction levels, we trained a M-RCNN with 3 segmentation classes, corresponding to ‘target’, ‘low-distractor’, and ‘high-distractor’. Low-distractors are non-target objects fixated by 1 or 2 people, while high-distractors are those fixated by 3 or more observers. The confusion matrices and computed metrics for one run of 5 fold cross validation can be seen in figure 22 and table IX. In general, this configuration causes a drop in the performance of the model. The MAPs decrease by 0.1-0.15, MARs decrease by 0.2-0.25, and F1-scores decrease by 0.09-0.19. Also, the network does not learn to detect low distractors. The accuracy of this class for car category is 0%, for bowl is 1.97%, and for bottle is 8.61%. High-distractors are learned better by the model. The accuracy of high-distractor class for car (46%) and bowl (43%) categories is higher than the bottle category (23.16 %). The target detection accuracy has also decreased a little compared to the previous configuration. Furthermore, the networks tends to miss more low-distractors and predict them as background compared to high-distractors. For bottle, 78% low distractors (191 out of 224) and 64% of high-distractors (114 out of 177) are missed. For bowl, 76% of low distractors (116 out of 152) and 0.52% of high-distractors (98 out of 190) are missed. For car, 0.71% (48 out of 64) of low-distractors and 0.44% of high distractors (38 out of 87) are missed. The network also confuses some low-distractors with high-distractors to a greater degree than vice versa. Moreover, the network often considers bigger objects as high distractors and smaller ones as low-distractors. This behavior is reasonable from a logical standpoint. In addition to a higher visibility associated with bigger objects, the bigger the area of an object is the higher the probability that a fixation lies on it. Figure 23 shows some example results of the network in this configuration.

To better understand the impact of our two-stage fine-tuning, which is composed of first fine tuning head layers and then all layers, we compare its performance against one stage training. We consider two variations, in one variation we only fine tune the head layers and in another we fine-tune all layers of Mask-RCNN network. Table X summarizes our results. In general, training only heads of the network has a higher performance compared to training all layers in one-stage fine-tuning. Only the target detection accuracy is slightly higher when fine-tuning all layers in one stage. We hypothesise that the features that are extracted for object-class instance segmentation on COCO dataset are useful for target-distractor segmentation on COCOSearch18, thus freezing those layers and only training the heads outperforms training all layers. Two-stage tuning still outperforms both one-stage versions on all computed metric. Although one-stage tuning takes less time and occupies less memory, the performance gain from two-stage tuning outweighs the costs.

We compare the performance of Resnet50 backbone vs. Resnet101 for bottle category in table XI. Resnet101 performs almost $\times 2$ better on all computed metrics.

To understand how augmenting the dataset with horizontal and vertical flipping affects the performance of the model, we compared model performance in 4 cases: 1-no augmentation, 2-just augmenting the first stage 3-just augmenting the second stage 4-augmenting both stages (the default case). The results are listed in table XII. Adding augmentation improves the results on all metrics. Also, the improvement achieved by adding augmentation to the second stage is higher than augmenting the first stage. The reason could be that the second stage has a higher architecture complexity as we train all M-RCNN layers, compared to the first stage where we only train the heads. Augmentation is a good strategy to avoid overfitting when training larger networks with small datasets. The computed metrics for augmenting only the second stage and augmenting both stages are quite similar, with slightly higher variance in the results of different runs for the first case.

The method we discussed here, with its relatively quick training and inference, is a useful way of segmenting target and distracting objects in images during visual search.

One drawback of our method is that we need to train a separate Mask-RCNN for each target category. One possible future direction is to unify these category-specific models into one model.

Another shortcoming of our method is that we only consider an object as fixated if the fixation location lies exactly on its segmentation. However, it might be the case that multiple objects are fixated by each fixation. This would need us to consider a radius (similar to the high-resolution fovea image which typically covers a region of 1.5° of the visual field) around each fixation location and consider all objects whose segmentation partially lies under this radius as fixated.

Our network often predicts bigger objects as more distracting. This behavior might be caused by the higher visibility of larger objects or our current configuration, as more fixations lie on the area of larger objects. However, if we consider a radius around the fixations, then at each fixation many of the smaller nearby objects are also attended and might be considered as distracting as the bigger object.

Furthermore, object-based visual attention has the assumption that all parts of an object are equally salient, or in our case equally distracting. This assumption often does not hold true. For instance, in one of the search images there is a big monitor displaying images that attract most observers attention. While these images are only placed on the monitor’s screen, we consider the whole monitor as highly distracting.

It is worth noting that visual search is similar to object detection task in computer vision. In fact in both of our methods, we employ pre-trained networks for object-detection task to predict visual search behavior of humans. State-of-the-art object detectors such as Mask-RCNN (and faster-RCNN) have object proposal networks that suggest the most probable regions that could contain an object. Similarly, when searching for an object, humans fixate at locations that are likely to contain the target object. The contextual information/gist (which is assumed to be computed pre-attentively), and prior knowledge about where an object should be located affect these distracting locations/objects.

In terms of internal architecture, a CNN-based object detector and visual attention theories share some similarities. For instance, CNN uses kernels (filters) each corresponding to a specific feature. Each filter attempts to find the locations in the image where the activation of that feature is maximized. Then during network training, the best filters (features) for detecting a specific target object are chosen. Feature integration theory [47] also suggests that in conjunction search, humans scan the scene in a serial way, merging different features of each fixated location until finding the target. Feature integration theory proposes that human brain possesses feature-specific neurons that activate in locations where those features are present. However, unlike a CNN where features are calculated across the whole image pixels; humans only perceive features around the fixated location in an image. This is due to the fact that humans’ eye receive high-resolution image in a small region within the fovea; hence, eye movements are needed to perceive different parts of the scene with high resolution. The other reason is the computational limitation of human brain, which makes it difficult to perceive the whole scene with high-resolution.

A difference our segmentation model has with RCNN object detectors, besides using fixated objects for training the network, is that it looks for two object classes: target and distractor. The target object can only be one object category, for example in case of searching for bottle, only bottles can be the target objects. However, any other objects categories could be considered as distractors, for instance: glass, chair, spoon, fork, etc. While, an RCNN aims at segmenting multiple object classes which correspond to different object categories.