GPS-GLASS: Learning Nighttime Semantic Segmentation Using Daytime Video and GPS data

Supervised training of semantic segmentation networks requires pixel-level annotation, which is laborious and time-consuming to obtain. Because ground-truth annotation is publicly available only for some limited domains, e.g., Cityscapes [2] for daytime road scenes and GTA5 [25] for synthetic scenes, unsupervised domain adaptation (UDA) has received significant interest. There have been several approaches [36, 19] to achieve the goal of UDA for semantic segmentation. However, these approaches are focused on reducing the domain gap between synthetic and real.

The existing datasets developed for semantic segmentation of road scenes are biased toward daytime scenes [2, 7], segmentation networks trained without considering UDA tend to fail in handling nighttime scenes. Consequently, efforts have been made to reduce the domain gap between daytime and nighttime scenes.

Motivated by the widely studied image style transfer [12, 13, 21], one can try translating daytime scenes to nighttime scenes. Earlier studies along this direction [26, 32] were overly interested in the auxiliary task of the image style transfer rather than the main semantic segmentation task. Sakaridis et al. [28] constructed the Dark Zurich dataset, which contains daytime and nighttime images that are coarsely matched with GPS information. These daytime and nighttime image pairs are helpful in guiding the semantic segmentation of nighttime scenes, resulting in many follow-up studies, including their guided curriculum model adaptation (GCMA). Sakaridis et al. [29] proposed an improved version of GCMA that uses depth and camera pose information. Wu et al. [39] proposed a multi-target domain adaptation network for nighttime semantic segmentation via adversarial learning. Xu et al. [43] proposed nighttime domain gradual self-training and patch-level prediction guidance methods. Gao et al. [6] proposed a correlation distillation approach for cross-domain between synthetic and real nighttime.

However, the above methods [28, 29, 39, 43] have not attempted to precisely align daytime and nighttime images because such an alignment can even be a more difficult task than semantic segmentation. Xu et al. [40] aligned the daytime and nighttime images using an additional optical flow estimation network. However, the above method requires additional datasets and training stages for the optical flow estimation network. We notice that the invaluable information of GPS is obtainable when constructing datasets such as Dark Zurich. Consequently, in this paper, we propose to use the GPS information of daytime and nighttime images to guide the correspondence matching for nighttime semantic segmentation network training.

Various studies are being conducted to find a matching point between two images e.g., stereo matching [24, 22, 44], optical flow estimation [5, 10], and semantic correspondence [17, 18]. In the case of learning-based optical flow estimation approaches, Dosovitskiy et al. [5] introduced an end-to-end optical flow estimation method with CNNs. Ranjan et al. [23] proposed a spatial pyramid network that predicts flow in a coarse-to-fine manner. Sun et al. [31] proposed a method of warping the spatial feature pyramid and calculating the cost volume from the warped features. Teed et al. [34] proposed a recurrent unit for gradual flow refinement, demonstrating high performance with fewer network parameters. Recent transformer [37]-based optical flow models [14, 41, 42] further improved the optical flow estimation performance.

Although the above methods have contributed to the development of optical flow estimation technology, they are still suffering from correspondence matching between different domains. Recently, Zhou et al. [48] estimated optical flow to match and align two images captured under different weather conditions. Zhang et al. [46] proposed a method to train an image translation network by warping images from different domains. Lee et al. [17, 18] introduced a model called SFNet that predicts bidirectional correspondence between different instances of the same object or scene category. Inspired by the superior performance of SFNet, we propose a dense corresponding matching method in different domains for nighttime semantic segmentation. A simple correlation layer trains the segmentation network to extract features that are invariant to the domain gap, e.g., between daytime and nighttime.

Our method involves a source domain $S$ and two target domains $T_{d}$ and $T_{n}$, where $S$, $T_{d}$, and $T_{n}$ correspond to Cityscapes (daytime) [2], Dark Zurich-D (daytime), and Dark Zurich-N (nighttime) [28] datasets in our case study, respectively. Note that only the source domain has ground-truth segmentation labels, and the two target domains are coarsely paired according to GPS locations. As shown in Fig. 2, our GPS-GLASS consists of a single weight-sharing relighting network ($G_{R}$), a single weight-sharing semantic segmentation network ($G_{S}$), and two discriminators ($D_{d}$ and $D_{n}$), where we used the same architecture of DANNet [39] for these network components. Let $I_{s}$, $I_{n}$, and $I_{d}$ denote image samples corresponding to $S$, $T_{d}$, and $T_{n}$, respectively. These images are fed to $G_{R}$ to make $G_{S}$ less sensitive to illumination changes [39]. The segmentation results are obtained as ${P_{s}}={G_{S}}\left({{G_{R}}\left({{I_{s}}}\right)}\right)$, ${P_{d}}={G_{S}}\left({{G_{R}}\left({{I_{d}}}\right)}\right)$, and ${P_{n}}={G_{S}}\left({{G_{R}}\left({{I_{n}}}\right)}\right)$. Only ${P_{s}}$ has its corresponding ground-truth segmentation labels ${P_{s}^{*}}$, and the other two results ${P_{d}}$ and ${P_{n}}$ are supervised by the pseudo-label.

Specifically, the proposed training framework called GPS-GLASS obtains the pseudo-label by estimating dense correspondence between the daytime and nighttime images, where the correlation layer is applied to the intermediate features of the segmentation network. In addition, since the dense correspondence between daytime and nighttime images can be inaccurate, GPS-GLASS obtains another pseudo-label by estimating dense correspondence between the daytime images and applying GPS-based flow scaling. By using the two different sources for acquiring the pseudo-label, GPS-GLASS trains the nighttime semantic segmentation network without any annotation from nighttime images. The details of GPS-GLASS will be explained in the following subsections.

We use five loss terms for GPS-GLASS: light loss $L_{light}$, semantic segmentation loss $L_{seg}$, adversarial loss $L_{adv}$, discriminator loss $L_{dis}$, and warping loss. Because we use the same loss functions defined in DANNet for the first four terms [39], we only detail the warping loss in this subsection.

We now have $P_{n\to d}$ and $P_{d\to n}$ and their confidence maps ${M_{n\to d}}$ and ${M_{d\to n}}$, which can be used to supervise the training of the nighttime semantic segmentation network. Note that $P^{{}^{\prime}}_{n\to d}$ and $P^{{}^{\prime}}_{d\to n}$ are integrated to $P_{n\to d}$ and $P_{d\to n}$ since only consistent predictions are used by the confidence maps. First, we use $P_{d\to n}$ for the pseudo-supervision of $P_{n}$. Specifically, the first warping loss term ${L_{d\to n}}$ is defined as follows:

where $H$ measures the cross entropy, $E_{o}$ denotes the one-hot encoding [39], $\mathbb{C}$ is a set of all semantic segmentation classes, $N_{p}$ is the number of pixels. ${{P_{n}}\left({{\bf{q}};k}\right)}$ represents the probability of the $k$-th object class at the position ${\bf{q}}$ of ${P_{n}}$. Note that the cross-entropy loss is measured only for the reliable prediction with ${{M_{d\to n}}\left({\bf{q}}\right)}=1$. Here, we define a set of ignore indexes, $\tilde{\bf{Q}}$, as follows:

where ${\mathbb{C}}_{dyn}$ is a set of dynamic semantic classes, including cars, people, etc. Then, ${{\bf{Q}}^{-}}$ in Eq. (10) is defined as ${{\bf{Q}}^{-}}={\bf{Q}}\cap{\tilde{\bf{Q}}^{c}}$. We found this special handling is necessary to prevent undesirable pseudo-supervision of dynamic object classes.

The second warping loss $L_{n\to d}$ is defined as follows:

where ${{\bf{P}}^{-}}$ is defined in a similar manner as ${{\bf{Q}}^{-}}$. In other words, $P_{d}$ is used as the pseudo-supervision of $P_{n\to d}$ for the nighttime segmentation network training.

The objective functions for the target daytime and nighttime domains, $L_{T_{d}}$ and $L_{T_{n}}$, and the source domain $L_{S}$ are defined as:

where $\mu_{1}$, $\mu_{2}$, $\mu_{3}$, and $\mu_{4}$ are empirically chosen as 0.01, 0.01, 1, and 1, respectively, following the baseline method DANNet [39]. In every training iteration of GPS-GLASS, we sequentially optimize $L_{T_{d}}$, $L_{T_{n}}$, and $L_{S}$ for daytime, nighttime, and source domains, respectively.

The Cityscapes dataset [2] includes 5,000 images taken in street scenes with pixel-level annotations for a total of 19 categories. In total, there are 2,975 images for training, 500 images for validation, and 1,525 images for testing. We used the training set as the labeled source domain $S$ in the GPS-GLASS training stage.

We implemented GPS-GLASS using PyTorch. The training was performed with a single Nvidia Titan RTX GPU. Following [1], we trained our network using the stochastic gradient descent optimizer with a momentum of 0.9 and a weight decay of $5\times 10^{-4}$. We used Adam optimizer [16] for training the discriminators with $\beta$ of 0.9 and 0.99. The initial learning rate of the generator and discriminators was set to $2.5\times 10^{-4}$ and then reduced to the power of 0.9 using the poly learning rate policy. For data augmentation, random cropping of the size $512\times 512$ was applied with a scale factor between 0.5 and 1.0 for the Cityscapes dataset, and random cropping of the size $960\times 960$ was applied with a scale factor between 0.9 and 1.1 for the Dark Zurich dataset. In addition, we applied random horizontal flips for training. We used PSPNet [47] as the segmentation network model, which has shown state-of-the-art performance in nighttime semantic segmentation. We pre-trained PSPNet on the Cityscapes dataset for 150K iterations using $L_{seg}$. Then, we set the batch size to 2 and trained the model for 100K iterations.

We found that careful selection of the temperature value $\alpha$ in Eq. (2) is important for correspondence matching. The performance evaluation on Dark Zurich-val for GPS-GLASSes trained with different $\alpha$ values is shown in Fig. 5. From this grid search of $\alpha$, we chose $\alpha=10^{4}$ in our experiments.

In order to demonstrate the effectiveness of individual components of GPS-GLASS, several modified models of GPS-GLASS were trained, and the best performances in Dark Zurich-val are reported in Table 3. GPS-GLASS without any pseudo-supervision serves as a naive baseline, which leads to the lowest mIoU of 24.68. Due to the static loss [39], DANNet achieved a 12.08% mIoU increase compared to the baseline. We applied other inter-intra pseudo-label mixing methods: taking the average of two pseudo-labels or taking the label with the higher probability for each pixel. For Dark Zurich-val, these two methods, denoted as avg and max in Table 3, increased the mIoU by 11.24% and 11.02%, respectively, which are worse than the performance improvement obtained using the confidence map (13.51%). Both warping loss terms, $L_{n\to d}$ and $L_{d\to n}$, were found to be essential compared to their single-use. In addition, because we obtained the integrated correlation volume by element-wise multiplication of the correlation volumes from the local and global features, we evaluated the performance obtained without using the local or global feature. The use of both features resulted in 0.74% or 1.52% higher mIoU compared to the single-use of the local or global feature, respectively. Last, because GPS-GLASS obtains pseudo-supervision from both intra-matching and inter-matching, we evaluated the performance without applying intra-matching or inter-matching and obtained 3.33% or 1.98% lower mIoU compared to GPS-GLASS, respectively.