Co-SemDepth: Fast Joint Semantic Segmentation and Depth Estimation on Aerial Images

MDE can be addressed by means of classical methods like Structure from Motion (SfM) [13] or through Deep Learning methods. SfM methods generally suffer from limited accuracy and scarce availability of feature correspondences [14]. Deep learning methods have been a way out of these low-vision challences, where the input to the DL architecture can be the whole (single) image or image sequences. Estimating depth from single images causes the scale ambiguity problem [15]. Some of the state of the art methods in this family are MonoDepth  [5], MonoDepth2  [16], Adabins  [4], and DPT  [17]. Estimating depth from image sequences improves the depth estimates compared to single image methods by making use of the temporal relation between input video frames [18, 3, 11, 19] to better understand the scale of objects.

The training of MDE networks can be realized in a supervised or self-supervised fashion. In supervised methods [20, 21, 22, 23, 11], depth maps ground truth should be provided to the network. The main challenge for these methods is the scarcity of available annotated datasets, especially in the aerial domain [12, 24, 25], that should cover different scenarios and environments. Some of the most used datasets in the literature are KITTI [26] for autonomous driving applications, and NYU [27] for indoor scenes. Self-supervised methods can be trained to predict the depth without requiring ground truth depth maps. They use as a self-supervision the nearby video frames  [28, 3, 29] or synchronized stereo pairs  [30, 5].

Open problems: Few MDE methods mentioned their inference time and no benchmarking was found in the literature comparing the inference time of different MDE methods. Very few methods were benchmarked on low-altitude aerial datasets [11, 31]. We address such challenges by benchmarking our method on the aerial dataset MidAir [12]. In this benchmark we use both depth evaluation metrics and inference time to compare our performance with other state-of-the-art.

Semantic segmentation is widely addressed in the literature, in particular in the autonomous driving field [6, 7, 8, 9, 10] using CityScapes [32] dataset for benchmarking.

Unlike depth estimation, semantic segmentation networks are normally trained in a supervised fashion. This can be realized using image-based or video-based methods. Some state-of-the-art examples of image-based methods are PSPNet [33], ICNet [6], ERFNet [7], Bisenet [9], and SegFormer [8]. Many video-based methods [34, 35, 36, 37, 38, 39, 40, 41] make benefit from the fact that there is a big overlap between successive frames by avoiding to repeat the semantic segmentation computation on each frame. They extract features of key frames and use feature propagation using light optical flow networks or interpolation techniques to pass the information to sequential video frames. This is done to speed-up inference time. However, the mIoU achieved using these methods is usually lower than image-based methods.

Open problems: few semantic segmentation methods [42, 43, 44] were benchmarked on low-altitude aerial datasets We benchmark our method on MidAir dataset for semantic segmentation. We also adapt the codes of other state-of-the-art methods and benchmark them on MidAir. In addition, we benchmark our method on Aeroscapes [43] dataset.

The idea of joint or multi-task deep learning architectures has been explored in the literature for various vision tasks. In [45], a joint architecture is implemented for image segmentation and classification, while in [46] they developed a joint network for motion estimation and segmentation. There are some works [47, 48, 49, 50, 51, 52] that addressed joint depth estimation and semantic segmentation. Most of the architectures used in these works have a common feature extraction part (in the form of deep convolutional network) followed by two separate branches dedicated for the prediction of depth and semantic maps. In [47], it was found that the shared feature extraction part helps in making the semantic segmentation branch benefit from the learnt depth features in the shared encoder and leads to slightly better results in semantic segmentation.

Open problems: None of the mentioned methods was benchmarked on aerial data, which incorporate very specif challenges in terms of variability of depth and semantic maps. We benchmark our method and the joint RefineNet [49] on MidAir dataset and we make the codes publicly available.

The architecture of M4Depth [11] is an adaptation of the standard U-Net encoder-decoder network trained to predict parallax maps to be then transformed into depth maps. The authors define parallax as a function of perceived motion, thus it can be seen as a general form of stereo disparity for an unconstrained camera baseline.

The network takes as input a sequence of $n$ video frames (we choose $n=3$) and the camera transformation $T$ between each two consecutive frames. At each time step $t$ the encoder, same as in Figure 1, takes a $t$-frame $I_{t}$ and extracts image features at different scales using its pyramidal structure. Each encoder level is composed of two convolutional layers and a domain-invariant normalization layer (DINL) [53] to increase the network robustness to varied colors and luminosity conditions.

Then, the decoder, see top part of Figure 1, takes the feature maps at different resolutions obtained by the encoder at time $t$, the features extracted from the previous frame ($t-1$), the parallax map predicted at time $t-1$, and the camera motion transformation $T_{t}$ to predict the parallax map of the current frame $I_{t}$. This parallax $p_{t}$ is then transformed into a depth map $d_{t}$ following the transformation proposed in [11]. Each level of the decoder is composed of a preprocessing unit and a parallax refiner. The preprocessing unit is responsible for preparing the input to the parallax refiner at this level and the parallax refiner is a stack of convolutional layers responsible for giving an estimate of the parallax map at each level.

Depth loss: The network is trained in an end-to-end fashion and a scale-invariant $L_{1}$ loss is used to compute the loss between the predicted depth map $\hat{d_{i}}$ and the ground truth one $d_{i}$. The loss is computed at each decoder level and then accumulated across all the levels:

where $M$ is the number of decoder levels and $N_{p}^{l}$ is the total number of pixels in the image at level $l$.

Inspired by M4Depth, we propose a similar architecture for semantic segmentation depicted in Figure 2. The encoder is a stack of multiple levels and has the pyramidal structure where the resolution of the feature map is decreased while proceeding forward through the encoder levels. The feature map predicted at each encoder level is passed to its corresponding decoder level. Each decoder level is composed of a preprocessing unit and a semantic refiner. The preprocessing unit prepares the input to the semantic refiner and the semantic refiner at each level gives an estimate of the semantic segmentation map at a specific resolution. The resolution of the semantic map is scaled-up proceeding forward on the decoder levels.

In Figure 3 we show the modules used in our architecture. The encoder at each level is composed of 2 convolutional layers. In the first level, DINL [53] is added after the first convolution to increase the network robustness to varied colors and luminosity conditions. ReLU activation is applied after each convolutional layer and the resolution is decreased by a factor of 2 after each level. The pyramidal structure of the encoder helps in extracting both coarse and fine (global and local) features from the input image.

The preprocessing unit at each decoder level is a pure computational unit with no parameters to be trained. It performs two operations:

• •

  It upscales the semantic map $S^{L-1}$ and the semantic features $f_{S}^{L-1}$ estimated from the semantic refiner of the previous level by a multiple of 2 to match the resolution of the current level.

• •

  It normalizes the feature map $f_{enc}^{L}$ received from the encoder.

Similar to the parallax refiner, the semantic refiner at each level is composed of a stack of convolutional layers. The last convolutional layer has a depth equals 4 (depth of the semantic features map) + N (the number of semantic classes). The output of the semantic refiner is a predicted semantic features map and an estimated semantic segmentation map. We apply Softmax activation on the predicted semantic segmentation map to obtain a probability score for each class on every pixel.

Different from M4Depth, our M4Semantic architecture works on single images. We removed the time dependency in the semantic decoder because this produced 2 times faster results than the one with time dependency with a slight drop in accuracy (this architectural choice will be discussed in Section V.E).

Semantic loss: the standard categorical cross-entropy loss is used on the predicted semantic maps at each level. The ground truths are resized using Nearest Neighbour interpolation to match the resolution of the predicted semantic maps at intermediate levels. Then, these losses are aggregated through a weighted sum:

where $N_{c}$ is the number of semantic classes, $p_{t}$ is the output softmax probability score for the target class and $p_{j}$ is the output softmax probability score for class j.

To merge the two previously described networks, we adopt a multi-tasking shared encoder architecture [47, 48, 49, 51, 52]. The depth estimation and semantic segmentation networks share the encoder part for feature extraction, but each of them has its own decoder for their corresponding map prediction. An overview of our joint architecture is in Figure 1.

Loss Function: Our joint network is trained in an end-to-end fashion. The loss function for our architecture is defined as:

We incorporated a weighting factor $w$ forcing the loss values for semantic to lie within the same range of the losses for depth, and thus ensuring a comparable contribution for the two losses during training.

MidAir [12] is a synthetic dataset collected using AirSim simulator [54] consisting of 420K forward-view RGB video frames captured at low altitude in outdoor unstructured environments with various weather conditions. It contains annotations of depth maps, semantic segmentation, surface normals, stereo disparity, and camera locations. Hence, this dataset is suitable for training and testing our joint Co-SemDepth architecture.

We adopt the train-test split used in [11], but we randomly select 8 trajectories to create validation data. In the evaluation, the depth values are capped at 80.0 meters. We resized images to a resolution of 384x384 instead of the original 1024x1024. In the original semantic annotation of MidAir, there are 14 semantic classes: Sky, Animals, Trees, Dirt Ground, Ground Vegetation, Rocky Ground, Boulders, Empty, Water, Man-Made Construction, Road, Train Track, Road Sign, and Others. Since several classes are visually indistinguishable and some of them very small, we mapped them to a smaller set of 7 semantic classes : Sky, Water, Land, Trees, Boulders, Road, and Others. Specifically, we considered Ground Vegetation, Rocky Ground and Dirt Ground as Land, and Animals, Empty, Train Track and Road Sign in Others.

Aeroscapes [43] is a real dataset collected using drones at low-mid altitude in various outdoor environments. It consists of 3,269 images with 80%-20% train-test split and resolution of 1280x720. Some of these images are captured with a forward-view while others are captured with nadir (top) view. This dataset contains only semantic segmentation annotation. For this reason, we could not use this dataset for the training of our joint architecture; because it requires annotation of depth, semantic, and camera locations. However, we used Aeroscapes for the training and testing of our single M4Semantic network. There are 12 semantic classes in Aeroscapes, namely: Background, Person, Bike, Car, Drone, Boat, Animal, Obstacle, Construction, Vegetation, Road, and Sky.

We adopt Adam optimizer with the default momentum parameters $(\beta_{1}=0.9,\beta_{2}=0.999)$ and a fixed learning rate of $10^{-4}$. We apply image augmentation of random rotation, flipping, and changing color (contrast, brightness, hue, and saturation) during training and we train with batch size of 3. In the computation of the joint architecture loss $L_{total}$, after inspection of the loss ranges for depth (range from 0.0 to $\sim$1.0) and semantic (range from 0.0 to $\sim$7.0), we selected a weighting factor $w=0.1$. After training, we choose the checkpoint that produced the best validation results for evaluation on the test set.

Our workstation has 16GB RAM, Intel core i7 processor and a single NVIDIA Quadro P5000 GPU card running CUDA11.4 with CuDNN 7.6.5 and Ubuntu operating system. Due to the memory limits of our workstation, we predict the depth and semantic maps at a resolution equal to half the input resolution and then apply Nearest Neighbour interpolation on the output maps to scale up its resolution to the original size. As reported in [55], decreasing the image resolution can slightly decrease the accuracy, however, it gives the advantage of reducing the computational runtime and memory footprint and these are critical factors for aerial robotics.

To quantitatively evaluate the depth prediction results, we consider the commonly used evaluation metrics in prior works [11, 3, 16]. These include the linear root mean square error (RMSE), the absolute relative error, and accuracy under a threshold. For semantic segmentation, we use the commonly used mean Intersection over Union $mIoU$ metric. The Inference Time (Inf. Time) is computed in milliseconds per frame (ms/f).

We conduct experiments to compare the performance of our joint architecture Co-SemDepth to the two single architectures: M4Depth and M4Semantic, see Table I. Each architecture was trained equally for 60 epochs.

We can notice that the accuracy values (in terms of depth and semantic metrics) of the joint architecture are close to the single ones. The inference time of the joint architecture (49.6 ms/f) is lower than the sum of the two single ones ($44.9+9.8$). Moreover, the number of parameters of the joint architecture (5.2 Million) is less than the sum of the two single ones (3.06 + 2.61 Million) by around 500K parameters. During inference, Co-SemDepth required only 6.2GB of GPU memory while running M4Depth and M4Semantic together required 14.6GB of GPU memory. This makes Co-SemDepth compatible to run on the microcontroller Nvidia Jetson TX2 that has only 8GB RAM and that is widely used in robotics hardware.

The above signifies that using our joint architecture Co-SemDepth is more effective in terms of computational time and memory footprint than using the two single architectures while achieving very close accuracies. The trade-off between accuracy and computational cost in the multi-tasking architectures was previously discussed in [56] and it can differ depending on the environment.

MidAir: We compare the performance of Co-SemDepth with other open-source state of the art methods (both single and joint architectures) in depth estimation and semantic segmentation, see Table II. For each method, we fix the input size to 384x384 and the number of training epochs to 60. Other parameters for every method were kept as the default. More details about the parameters used for each baseline method can be found on our Github page.

We can clearly notice that our method outperforms the other joint network, RefineNet, in both semantic and depth accuracies and inference time.

Compared to the single depth estimation networks, Co-SemDepth could maintain its superior accuracy in depth estimation. This indicates that transforming M4Depth to the joint Co-SemDepth did not have a negative effect on its depth estimation performance compared to state-of-the-art. Co-SemDepth also has a notable fewer number of parameters compared to the other methods.

For the single semantic segmentation networks, Co-SemDepth has a competitive mIoU with the others, only slightly inferior to ERFNet, which is in any case a dedicated architecture.

The per-class IoU evaluation can be found in Table III and a qualitative visualization of the semantic map predictions of three different methods (ERFNet, MobileNet, and Co-SemDepth) can be found in Figure 4. We can notice that the three methods could capture the overall semantic layout of the input images. However, Co-SemDepth and ERFNet are remarkably better in capturing the details (notice the trees and the train track) than MobileNet. Compared to Co-SemDepth, ERFNet is a bit better in capturing some of the faraway details (the distant boulders and bushes in the third row).

Aeroscapes: We report the M4Semantic results in Table IV. Our network was trained for 200 epochs with a batch size of 3 and a learning rate of $10^{-4}$ for the first 70 epochs and then decreased to $10^{-5}$.

We implemented M4Semantic on TensorFlow as one whole model that can be trained in an end-to-end fashion without separation between the weight files of the encoder and the decoder. This led to a more compact code but limited us from pretraining the encoder separately on Imagenet as done in the other methods. Nevertheless, we could produce a competitive mIoU compared to the others. An output visulization can be found in Figure 5.

We conduct architecture study experiments on M4Semantic to highlight the importance of the addition or ablation of different modules, see Table V. From the top part of the table, we can notice that using 5 levels produced the best mIoU.

In the bottom part, Original means M4Semantic(5 level). In Original+{SNCV}, we used the Spatial Neighbourhood Cost Volume module used in the decoder in M4Depth [11] on the encoded feature maps instead of adding the normalized feature maps directly in the preprocessing unit, Figure 3. Such a module measures the two-dimensional spatial autocorrelation of the scene and improved the performance in depth estimation. However, as can be seen in the table, using such module in M4Semantic didn’t improve the performance in semantic segmentation and, moreover, it increased the inference time.

In Original+{$S^{t-1}$}, we test the addition of time dependency in M4Semantic. At each decoder level, the semantic segmentation map predicted of the previous frame $S^{t-1}$ is used along with the camera motion information and the ground truth depth map to warp it and give an initial prediction of the semantic map of the current frame. We concatenate such warped map with the output of the preprocessing module, Figure 3, at each decoder level. While such technique achieved higher mIoU, the inference time increased due to the added warping computation. Also, the warping of the semantic segmentation map requires the ground truth depth map and this is not available in most of the times in reality.

Given this architecture study and the one done on the single depth estimation architecture [11], we choose the number of levels of our joint architecture Co-SemDepth to 5 levels.