Multi-source Pseudo-label Learning of Semantic Segmentation
for the Scene Recognition of Agricultural Mobile Robots

Semantic segmentation is a task to predict an object class on each pixel of an image. It is one of the most important tasks in the field of computer vision and its application includes scene recognition for the navigation of self-driving cars and autonomous mobile robots. After Long et al. [9] proposed the idea of converting classification networks into a fully convolutional network to produce a probability map for the input of arbitrary size, most of the semantic segmentation models have been developed based on it [10, 11]. Many recent high accuracy models use ResNet proposed by He et al. [12] as a backbone network to build very deep networks [13, 14]. While it is proven to be effective to stack many layers to get a high accuracy, the performance comes at the high computational cost. To tackle this problem, computationally efficient models have also been studied, targeting the applications such as robotics and autonomous driving cars where real time recognition is required for the operation [15, 16, 17, 18].

Image segmentation techniques have been studied for navigation tasks in agricultural fields. Lulio et al. [19] proposed an image segmentation method for robot navigation in orchards with a combination of color-based region segmentation and an artificial neural network. Sharifi et al. [20], Aghi et al. [21], and Lin et al. [22] proposed DNN-based segmentation methods for the purpose of robot navigation.

In contrast to “shallow” methods, the DNN-based methods have several advantages. First, the generalization performance of DNNs is better than that of the image processing methods and machine learning methods with hand-crafted features. Second, the trained DNNs provide discriminative “deep” features that can be used in other tasks [23]. In fact, we used the features learned in the proposed method to estimate the traversability of the objects and showed that the features are capable of discriminating between traversable and non-traversable plants [2].

Moreover, unlike other DNN-based methods, our proposed methods do not require any image datasets with manually provided labels dedicated for the target environment, i.e., greenhouses.

Semantic segmentation based on deep learning usually requires a large amount of training data labeled on each pixel and manual labeling is especially time-consuming. One promising approach is to use a large amount of image-label pairs automatically synthesized by computer graphics. Since the appearance of the synthetic images is different from real-world images, the model trained on the synthetic data does not perform well in the real world. Therefore, we need UDA to re-train the model in a different domain i.e., the real environment. There are two major approaches to UDA methods [24, 25]. One is called “domain alignment” where a model is trained so that the distribution discrepancy of domains is minimized. The alignment is done in different levels such as the input space [26, 27], feature space [28, 29], and output label space [30, 31]. Those methods aim to learn domain-invariant features of both domains. Recently, such training is often realized by adversarial training [32, 29].

The other approach of UDA is pseudo-label-based methods. This type of method uses outputs from a model pre-trained with a source dataset as true labels. By directly training the model on the target images with the pseudo-labels, it can fully learn domain-specific information. However, the pseudo-labels for the target images inevitably contain misclassification and it may affect the training as noise. To deal with the noise, different approaches have been proposed, such as thresholding based on the confidence of the prediction [33, 34], the adaptive weighting of loss values based on the uncertainty of the prediction [7], and prototype-based pseudo-label denoising [8].

Apart from the aforementioned single-source adaptation methods, there are several studies of multi-source domain adaptation (MDA). Mansour et al. stated that the target distribution can be represented as a weighted mixture of multiple source distributions. Following this analysis, Xu et al. [35] implemented an MDA training with deep networks. For a detailed survey of MDA, readers are referred to Zhao et al. [36]. Zhao et al. [37] extended Cycada [26] to MDA for semantic segmentation. A work of Nigam et al. [38] utilizes multiple models trained with different source datasets as feature extractors, and adapt the segmentation layers to a target dataset of overhead images from a drone. Although their method is similar to ours in a way that multiple pre-trained networks are exploited, their task is supervised domain adaptation where they prepare their own labeled dataset, while our method utilizes publicly available datasets that are not specifically built for greenhouse scenes.

As a benchmark of semantic segmentation, a task considered de-facto standard is adaptation from synthetic datasets such as the GTA5 dataset [4] and the SYNTHIA dataset [5] to a real dataset, most often Cityscapes dataset [6]. In this task, scenes are limited to urban scenes in both the source datasets and the target datasets. Besides, the set of object classes is identical in all the datasets. In this paper, we further investigate the methodology of transferring knowledge in cases where the structure of the environments is different, and the source and the target datasets do not share their label space.

Our intuition behind this method of multi-source pseudo-label generation is that while the prediction from each source model contains a lot of misclassifications, the models correctly predict many of the pixels in common. Fig. 3 shows outputs for a target image from different source models trained with the CamVid dataset [39], Cityscapes dataset [6] and the Freiburg Forest dataset [40] as well as the target image, hand-labeled ground truth, and the pseudo-label generated from the three outputs. As can be seen in Fig. 3, many of the true regions of “plant” class and “ground” class are correctly classified in all the models. This implies that getting a consensus of the models is effective for generating precise pseudo-labels as shown in Fig. 3. Another thing worth noting is that each model has its own characteristics of prediction. For example, the prediction of the model trained with Freiburg Forest dataset on the ground regions better captures the shape of true ground regions than the other two models. However, the prediction on the plants and artificial objects is not accurate. On the other hand, in the model trained with Cityscapes, the prediction on the ground region includes a lot of false positives, while that on the other objects is more accurate than the Freiburg Forest model. These differences stem from the structural differences of the source datasets. The proposed method avoids biased training to the structural features of a specific source dataset dissimilar to the target by getting agreements of multiple source models.

In this step, pseudo-labels for the target images are generated using outputs from the models pre-trained with the source datasets. First, the target images are fed in all the pre-trained models to yield predicted labels in the label space of each source domain. The predicted label from the model of source $i$ for the $j$th target image is given as follows:

The predicted labels are then mapped to the target label space using a label conversion function $\xi_{i}:\mathcal{Y}_{i}\rightarrow\mathcal{Y}_{T}$, where $\mathcal{Y}_{i}$ and $\mathcal{Y}_{T}$ denote a set of labels in source $S_{i}$ and target $S_{T}$, respectively:

Here, we heuristically define the mapping from the source classes to the target classes for each source domain as shown in Table 1. After the label conversion, for each pixel of the target images, a predicted label is assigned to the pixel if the prediction is the same among all the source models. Otherwise, “other” label is assigned and not used for training. Formally, a pseudo-label for the pixel $(h,w)$ of the $j$th target image $x_{T,j}$ is generated as follows:

where $c_{k}\in\mathcal{Y}_{T}$ is a class category in the target dataset and $c_{\phi}$ is a class label “other” that is not used for loss calculation during the training and thus does not affect the training.

In standard training of semantic segmentation models, the cross entropy loss is widely used. In the training with pseudo-labels, however, equally treating all the pseudo-labels may lead to poor performance due to noise in the pseudo-labels. Zheng et al. [7] proposed an adaptive loss weighting that decays as the uncertainty of the estimation increases so that the losses on the pixels with high uncertainty do not affect the training much. Their method exploits an auxiliary segmentation branch to predict pixel-wise uncertainty of the label prediction. We attach an auxiliary segmentation branch to ESPNetv2 for uncertainty estimation. The entire segmentation network is shown in Fig. 7 in Appendix 6.

Given the output of primary branch $F(x;\theta)$ and that of the auxiliary branch $F_{aux}(x;\theta)$, the uncertainty of the prediction is defined as Kullback-Leibler Divergence (KLD) of the distributions of class probabilities from the two branches:

Using eq. (6), the loss function is defined as follows:

In the middle of the training, we update the pseudo-labels with the outputs from the current model to further leverage the knowledge learned from the initial pseudo-labels. We introduce pseudo-label updating with prototype-based denoising inspired by Zhang et al. [8].

At first, the prototypes are calculated. The prototype of class $c$ is a mean of the features that belong to the class $c$ and is calculated using the current pseudo-labels as follows:

where $f^{(h,w)}$ is an intermediate feature provided by the currently trained model $F\left(\cdot;\theta\right)$ at pixel $(h,w)$, and $\mathbbm{1}$ is the indicator function that returns 1 when its argument is true and 0 otherwise.

Pseudo-labels for image $x_{T,j}$ are then updated as follows:

where $\mu=\sum_{c^{\prime}}\omega^{(h,w,c^{\prime})}p_{j}^{\left(h,w,c^{\prime}\right)}$ is a normalization factor and $\alpha$ is a threshold for pseudo-label selection, which is set to 0.9 in the experiments. $\omega$ is a vector of weight values to modulate the class probabilities, calculated using the distances between the feature and each prototype. For each pixel, the weight $\omega$ is calculated as follows:

where $\tau$ is a temperature parameter that controls the degree of bias of the distribution, which is set to 1 in [8]. The values of $\omega$ indicate the class likelihood of the feature based on the distance from the feature to each prototype, and the probabilities predicted by $F\left(\cdot;\theta\right)$ are rectified by $\omega$.

We use PyTorch implementation of ESPNetv2 [16]. For estimating uncertainty, an auxiliary segmentation branch is attached. The architecture is shown in Fig. 7 in Appendix 6. The processing time was about 44 [msec/image], or 23 [fps]. The number of floating point operations (FLOPs) is 0.79 [GFLOPs] while that of the original ESPNetv2 for semantic segmentation is 0.76 [GFLOPs] [16], indicating that attatching the auxiliary branch did not affect the computational efficiency. All training of the proposed method is performed on one NVIDIA Quadro RTX 8000 with 48GB RAM. The numbers of training rounds and epochs per round are set to 10 and 5, respectively. The learning rate is fixed to $5\times 10^{-4}$ and the batch size is 48. Adam [41] is used as an optimizer. Weights of the target models are initialized by the supervised training of semantic segmentation on the CamVid dataset [39], except for the last classification layer initialized with random values. We measure segmentation performance with the Intersection over Union (IoU) metric.

We use CamVid [39], Cityscapes [6] and Freiburg Forest dataset [40] (hereafter referred to as CV, CS, and FR, respectively) for the training of the source models. CV, CS, and FR have 367, 2975, and 230 labeled training images, respectively. Label conversions from the source datasets to the target datasets ($\xi_{i}$) are heuristically defined as shown in Table 1 . A more detailed analysis of the label mapping is in Appendix 7.

As target greenhouse datasets for adaptation, we acquired images from three greenhouses; Greenhouse A that grows tomatoes, Greenhouse B that is the same greenhouse as Greenhouse A but acquired in different date and time, and Greenhouse C that grows cucumbers. Each target dataset has 6689 unlabeled images. Example images of the target datasets are shown in Fig. 4.

For each target dataset, we prepared test images with manually annotated labels for evaluation. The numbers of the test images are 100 for Greenhouse A, and 50 for B and C. The labels on the test images include the “other” class and the pixels with the class are not considered in the calculation of the metric (See Fig. 5). More detailed annotation policy is described in Appendix 8.

To evaluate the relative performance of our proposed method, we conducted training with existing methods of supervised learning as well as single-source and multi-source UDA for semantic segmentation. Implementation details of the baseline methods are described in Appendix 10.

For evaluation of the supervised baselines, we conducted 10-fold cross-validation with the 100 labeled images of Greenhouse A, which are used as the test data in the experiments of the UDA methods. The reported results are averages of IoUs on the test fold of each of the 10 training trials. Note that the training setting and the evaluation condition are different from the UDA training. This is due to the limited amount of manually labeled greenhouse images available. Nevertheless, we report the results of the supervised baselines to demonstrate baseline performances that we can expect from supervised learning.

We used the source code distributed by the authors with modifications to apply them to our experiments, such as adding the new datasets and converting the source labels to the target label set defined in Table 1. In terms of ProDA, we report the result of stage 1 proposed in [8]. Although the original method employs knowledge distillation after stage 1 to boost the performance, they reported the state-of-the-art performance at the time with only the training of stage 1. We, therefore, suppose the result of stage 1 on our setting is sufficient as a baseline. The rest of the training process follows the algorithm of each method. Implementation details are in Appendix 10.2.

Although there are also multi-source UDA methods such as Multi-source Domain Adaptation for Semantic Segmentation (MADAN) [37], we could not train it due to limitations of computational resources. In Appendix 11.2, we show results of image style transfer between the source dataset and the target greenhouse dataset, which is the first step of MADAN.

In the comparative studies of single-source UDA methods, we report the results on Cityscapes (CS) since it provided best results than CV and FR) in all of the single-source baseline methods. We suppose this may be because CS has a sufficient amount of images (about 3000) for training models with a large capacity such as DeepLab v2, while the other source datasets (CV and FR) have only a few hundreds. On CV and FR, the model may have overfit to the training sets due to the scarcity of the data, and thus resulted in poorer adaptation performances. As there was no significant difference of adaptation performance between the source datasets in the single-source setting shown in Sec. 4.2 (see Table 2), we believe it is reasonable to use CS to demonstrate the performance of the baselines on behalf of the three source datasets.

Table 3 shows the results.

Overall, the proposed method outperformed the baseline methods except for SP-DNN, and thus exhibited the ability to transfer knowledge from the datasets dissimilar to the target dataset. Although the performance of the baseline methods might be improved with extensive hyper-parameter search, the results indicate that relying on a single source is not effective enough even when using state-of-the-art UDA methods in our task where the source and target datasets do not share their structure of the scenes. Qualitative results are shown in Appendix 11. While SP-DNN resulted in the best IoUs, it also showed a clear tendency of overfitting due to the scarcity of training data. Although increasing the training data will resolve the problem, it requires laborious manual annotation. On the other hand, the proposed method enables training with a large amount of unlabeled images and provides a comparative performance.

Among the baseline UDA methods, CRST resulted in better performance than others, although they are reported to be better than CRST [7, 8]. We conjecture that adversarial learning negatively contributed to these results. ProDA and Seg-Uncertainty involve adversarial learning in the training processes to make the spatial structure in the output space as close as possible between the source and the target datasets [30]. Adversarial learning, including GANs [32], is effective when the difference between the source and the target stems mainly from the styles such as texture and color and the datasets share similar structure. For instance, GANs can be used to transfer the image style of source images closer to the target images without significantly changing the image contents, i.e., semantic class of corresponding image regions, so that a segmenter for the target images can be trained on the transferred source images in the target’s style with the ground truth labels. However, when the structure of the scenes is different, adversarial learning does not necessarily convey meaningful information. In fact, as shown in Appendix 11.2, the image contents are not preserved between the original and the translated images after CycleGAN-based image style transfer between CS/FR and Greenhouse A. For example, some ground regions in CS are transferred to green plant-like stuff, and some plant regions are transferred to regions with sky-like appearance (See Fig. 11). In [37], the authors also pointed out that the structural differences of objects between the source and the target datasets led the training to poor performances on those object classes (e.g., the source in target images are much taller than those in target images). Those facts imply adversarial learning is not as effective for adaptation under such structural differences between the source and the target scenes.

DeepLab v2 [48] with ResNet101 backbone [12] is used as the semantic segmentation model in the original work of all the baseline methods of UDA. Although it is different from our model, we also adopt DeepLab v2 in the baselines to keep their implementation as much as possible. CRST and ProDA are trained on an NVIDIA Quadro RTX 8000, and Seg-Uncertainty is trained on an NVIDIA GeForce 1080Ti with 11GB memory.

In CRST [34], the source models are pre-trained by supervised training and we follow the implementation. We first train a model with a source dataset with its original label set for 200 epochs. We then replace the final classification layer with $C$-class classifier where $C$ denotes the number of object classes in the target dataset. The model with the replaced classifier is fine-tuned with the source labels converted to the target label set defined in Table 1 for 50 epochs. Initial learning rate is $1\times 10^{-4}$ for the final classification layer and $1\times 10^{-3}$ for the rest of the layers. The polynomial learning rate scheduling [49] is used with the power of 0.9. Other hyper-parameters are unchanged from their source code.

We train Seg-Uncertainty [7] with MRNet [50]. MRNet first trains, as a warm-up, a segmentation network with both the source and the target datasets via adversarial training at the segmentation output which was originally proposed in [30]. Other hyper-parameters are unchanged from their source code.

ProDA [8] follows the same warm-up strategy as Seg-Uncertainty. We, therefore, use the same warm-up models trained in Seg-Uncertainty in ProDA. Although ProDA and Seg-Uncertainty both use DeepLabv2 as a segmentation network, the architecture of the segmentation networks is slightly different between Seg-Uncertainty and ProDA, such as the number of channels of the intermediate feature map. We modified the source code of ProDA to adjust to the network. Other hyper-parameters are unchanged from their source code.