BEV-MODNet: Monocular Camera based Bird’s Eye View
Moving Object Detection for Autonomous Driving

To be able to project a scene from image view into BEV, one would need 8 points to perform such operation. Four points have to be determined in the source frame and the corresponding 4 points have to be identified in the target frame. A homography matrix is computed to transform the source image into the target image, which is usually done in an iterative approach. For the application of autonomous driving, one interesting feature would be the lanes of the road. They can be used so that they become parallel to each other in the final projection. Due to the lack of 3D information, the projected image is not realistic and there is a lot of noise in the output. Figure 3 demonstrate a sample image of KITTI dataset when projected on BEV using IPM [12] in (d). We also evaluate predicting MOD using image view and then doing projection on top view using IPM in (c). As observed in the image, (c) provides very noisy output relative to (b) which is the representation we would like to learn end-to-end. To learn such representation directly using a deep network, one would need a dataset with such representation which is not available in the public datasets. Hence, we created our own dataset having MOD annotations on BEV.

In general, there is a limitation of large scale MOD datasets in autonomous driving. In [25], 255 images on KITTI dataset have been manually labeled for motion segmentation task. Additionally, around 3k images on Cityscapes dataset have been annotated. These numbers are relatively low, and they are only performed on image view of the camera sensor. In [22], 1.3k images have been weakly annotated for MOD, and it has been extended by [18] where more KITTI sequences have been annotated for moving vehicles only. All these methods provide only image view annotation which cannot be used directly on top-view predictions. Hence, we create our own dataset which consists of 12.9k images including pixel-wise annotations for static and moving objects. Our dataset is labeled for 5 classes, and it will be released for all classes, however in our experiments we focus on training the network with vehicle class only to simplify the problem. We choose KITTI dataset because of the extensive prior work on MOD to enable comparison. Another alternative is NuScenes [2] dataset. It is a more recent larger dataset which provides motion attribute for vehicles and pedestrians but not for cyclists and motorcyclists.

We adapt the approach by [18] to create our dataset, where we make use of KITTI raw sequences as they provide IMU/GPS measurements for ego-motion, LiDAR point clouds for depth and 3D boxes of the objects relative to the ego-vehicle in each frame. First, we use the IMU/GPS to compute the ego-vehicle motion in LiDAR coordinates system. We use the tracking information to compute the difference in objects positions between each two sequential frames. We project the objects into world co-ordinate system and we compute the difference between ego-vehicle and other objects motion. Using thresholding techniques, we are able to classify the surrounding objects into moving or static ones. We project the 3D points in the 3D camera co-ordinate system onto the $xz$ plane to obtain the BEV representation of the surrounding scene.

At first, we obtained the max distance of the furthest object in the dataset and set it as maximum distance to make sure all objects are included in the dataset. However, we observe that most of the objects are closer to the ego-vehicle and most of the output maps are mostly empty because of the sparsity of very far objects. Hence, we keep the maximum distance to $50m$ to maintain reasonable resolution of the output maps. We observe that there are still false positives and negatives in our moving/static labels due to thresholding errors. We fix such errors by manual refinement of the false labels. The main task we target is motion segmentation, so we provide the annotations as pixel-wise masks for moving objects. We also provide masks for static objects which may be used to improve the motion segmentation prediction. Figure 2 represents samples of our generated dataset. The blue boxes refer to static objects from BEV viewpoint and the red ones correspond to moving ones. Table I demonstrates details about our generated dataset. We provide annotations for 5 classes and we provide details about both static and moving objects in each class.

As proposed by [13], an encoder-decoder architecture is able to learn a BEV representation of the surrounding scene. We follow a similar approach and make use of the architecture in [10] for its high inference rate and low weight. The model in [13] is able to predict BEV map without focusing on moving vs static object classification using a single monocular image. On the other hand, the model in [10] takes two sequential images as input and tries to understand motion implicitly. This approach has been proven by [16] that it provides less accuracy than feeding optical flow explicitly. We create another encoder which accepts optical flow as input and we perform feature fusion [17] between multi-scale feature maps of both encoders and then feed the resulted tensors into the decoder. The decoder consists of 5 deconvolution layers which are preceded by convolution layers [4]. The final output is a binary mask which predicts a class for each pixel among the two classes (Moving and Static).

We use ResNet18 as backbone for feature extraction, where we create another encoder for capturing motion features from optical flow. We initialize our network with the ResNet18 pre-trained weights and we set the batch size to 16. The network is trained using the Ranger (RAdam[11] + LookAhead [28]) optimizer. We train all the models using weighted binary cross-entropy loss function for 60 epochs. We use transposed convolution layers for upsampling purpose to finally reach the original input size. Finally, a weighted binary cross entropy loss function is used to obtain the final predictions for each pixel as a classification task among two classes, i.e, Moving and Static.

Table III demonstrate the results using our baseline network to predict BEV MOD end-to-end vs doing the prediction on Front view and performing IPM afterwards. We evaluate both predictions vs our generated ground truth and we obtain significant improvement over IPM approach by approx 7% in mIoU. Figure 5 demonstrates the output of our network in 2nd row vs the ground truth from our dataset in 3rd row. It is shown that CNN is able to predict BEV directly through an encoder-decoder architecture as proposed by [13]. However, the network is not able to distinguish between vehicles in some cases as described in the first column. In the second column, the object highlighted in yellow is very hard to predict where most of the object is occluded behind the front vehicle. On the other hand, the static object highlighted in blue has been suppressed correctly which shows the importance of optical flow to capture the motion information in the scene. The third and fourth columns demonstrate our results for multiple objects in the scene. Results show decent output where the model can be used a baseline for benchmarking. However, the length of some objects are not captured perfectly and some of the objects are mis-classified as false static objects, which shows that there is still room for improvement. Perhaps one approach to explore is to define an explicit learning-based BEV projection model inside the network to overcome such inaccuracies.

We also observe that accuracy of the predictions decrease with increasing the depth from the camera sensor. To evaluate that quantitatively, we divide the view range into 5 bins, 10m each, and we compute mIoU over all the images for the 5 bins separately, where the results are tabulated in Table III. We observe that very close objects are not captured entirely and this is expected because very close moving objects are usually moving in the same direction with almost same speed of the ego-vehicle such as passing vehicles. Due to motion parallax problem, such vehicles appear almost fixed relative to the ego-vehicle and hence very hard to detect. Accuracy of the model reaches its maximum in the middle ranges from 10m to 30m from the camera sensor and then decrease again as we go far away from the sensor. This is intuitive because as we go far away from the camera sensor, the vehicles appear smaller and moving slowly from the viewpoint of the ego-vehicle. This makes the optical flow vectors associated with such vehicles are small compared to the closer vehicles, and hence they are harder to detect. Moreover, due to absence of depth sensor, ambiguity increases with increased depth which might cause inaccurate predictions. We also observe that using the same input modalities and same architecture, the network learns motion segmentation in front view in a better accuracy. This is expected because in BEV approach the network learns an additional task to motion segmentation which is BEV projection. However, overall, BEV end-to-end learning provides better accuracy than learning in front view then projecting using IPM as demonstrated in Table III. Using more complex models than our model, better representation can be learnt using our dataset with higher accuracy.