Cyber Mobility Mirror for Enabling Cooperative Driving Automation in Mixed Traffic:
A Co-Simulation Platform

To model the evolution of traffic states based on traffic dynamics and interactions between traffic objects, microscopic traffic simulators have been developed for decades and greatly stimulated the development of intelligent transportation systems [10]. These simulators mainly consist of three components: 1) transportation network defining road topology; 2) traffic generator creating traffic flows with certain demand distributions; and 3) microscopic traffic flow control strategies, including traffic signal management, vehicle driving behaviors, and moving strategies for pedestrians.

Several simulation platforms are of great popularity in CDA research, such as VISSIM [11], Aimsun [12], and SUMO (Simulation of Urban MObility) [13]. Specifically, VISSIM and Aimsun are widely used in dealing with multi-modal traffic flow simulation due to their capabilities of providing fundamental 3D preview and statistical simulation results. SUMO is an open-source, highly portable, microscopic, and continuous traffic simulation package designed to handle large networks [7]. Additionally, high compatibility to connect and interact with different kinds of external simulators, e.g., OMNeT++ [14], CARLA, etc., is one of the key features of SUMO. These microscopic traffic simulators mainly focus on general assessment for traffic dynamic performance at the network level under different traffic scenarios. Nevertheless, the design and assessment of CDA-based cooperative perception, decision-making, or control models highly rely on the fidelity of sensor data, which is a major challenge for these conventional simulators. In recent years, simulators that are capable of modeling high-fidelity sensors gain more and more interests, which are introduced in the following section.

With the development of CDA, especially autonomous driving technologies (ADTs) [15], the requirement for high-fidelity sensors in simulators has gained more and more attention. In recent years, several autonomous driving simulators quipped with high-fidelity sensors have been developed based on game engines, such as Unity [16] and Unreal Engine [17]. For instance, AirSim [18], SVL, and CARLA have the capability to offer physically and visually realistic simulations for autonomous vehicle technologies (AVTs) as well as CDA systems. Specifically, AirSim includes a physics engine that can operate at a high frequency for real-time hardware-in-the-loop (HIL) simulations with support for popular protocols, such as MavLink [19]. SVL is a high-fidelity simulator for AVTs, which provides end-to-end and full-stack simulation that is ready to be hooked up with several open-source autonomous driving stacks, such as Autoware [20] and Apollo [21]. CARLA, an open-source simulator for autonomous driving, supports flexible specifications of sensor suites and environmental conditions. In addition to open-source codes and protocols, CARLA provides open digital assets (e.g., urban layouts, buildings, and vehicles) that can be used in a friendly manner for researchers.

These simulators have been developed from the ground up to support the development, training, and validation of AVTs, enabling the development of CDA. They have the capacity to assess the CDA system in a cost-effective manner as well as to provide high-fidelity sensing information.

Although having these existing simulators, researchers still get struggled with the imperative assumption that perception data (e.g., location, velocity, rotation, etc.) is collected directly from intrinsic attributes of simulation engines, when they develop and evaluate their high-level CAV functions, such as decision-making or control methods for CDA. Therefore, to develop a generic platform that can not only support physically and visually realistic simulation but also provide perception data based on high-fidelity sensor information is still a research gap for enabling CDA system research and development.

At the early stage of traffic detection, sensors with low-computational power are widely used, such as loop detectors, radar, ultrasonic, etc. Although most of them are still implemented in contemporary transportation systems, they suffer from different kinds of innate problems, such as detecting uncertainties, traffic disruption at installation, and high maintenance costs [23]. With the development of computer vision (CV) technology and the improvement of computational power, camera-based traffic object detection has been widely developed. Aslani and Mahdavi-Nasab [24] tried to gather useful information from stationary cameras for detecting moving objects in digital videos based on optical flow estimation. In situations where limited memory and computing resources are available, Lee et al. [25] presented a moving object detection method for real-time traffic surveillance applications based on a genetic algorithm.

Recently, LiDAR sensors is increasingly implemented for traffic object detection tasks due to their advantage of having higher tolerance of lighting conditions and accuracy of relative distance. Regarding traditional methods to deal with the 3D point cloud, one popular workflow is 1) background filtering; 2) traffic object clustering; and 3) object classification [26]. Additionally, the LiDAR point cloud can also be used to identify lane markings [27, 28]. Although some traditional methods have been applied to the 3D point cloud, the greater potential of LiDAR data should be tapped by data-driven methods (e.g., deep learning) which are introduced next.

The development of deep neural networks (DNNs) has significantly improved the possibility of dealing with large-scale data, such as high-fidelity images or 3D point clouds. With the great success of deep learning in the image recognition area [29, 30], many DNN-based models have been implemented in object detection for traffic scenarios using cameras or LiDAR sensors. Chabot et al. [31] presented an approach called Deep MANTA, for multi-task vehicle analysis based on the monocular image. In terms of different lighting conditions, Che-Tsung Lin developed a nighttime vehicle detection method based on image style transfer [32]. Chen et al. [33] proposed a shallow model named Concatenated Feature Pyramid Network (CFPN) to detect smaller objects in traffic flow from fish-eye camera images.

Additionally, 3D-LiDAR is also getting more popular in traffic object detection [34]. For instance, Asvadi et al. [35] presented an algorithm named DepthCN which used deep convolutional neural networks (CNNs) for vehicle detection. Considering the real-time requirement for autonomous driving applications, Zeng et al. [36] proposed a real-time 3D vehicle detection method by utilizing pre-RoI-pooling convolution and pose-sensitive feature maps. Simon et al. [37] proposed an Euler-Region-Proposal for real-time 3D object detection with point clouds, called ComplexYolo, which is capable of generating rotated bounding boxes for 3D objects. For CDA applications, traffic object detection needs meticulous consideration. Thus, in our CMM co-simulation framework, the ComplexYolo model is adapted with customized improvements for real-time vehicle and pedestrian detection.

CARLA has provided several well-developed virtual towns with different road maps and textures. In this paper, we implement “town03” as our fundamental traffic map and select a target intersection for research. This is shown in the left part of Figure 3. Additionally, we can generate vehicles and pedestrians via CARLA-based python scripts. The right part of Figure 3 shows a specific top-down view of the traffic scenario with vehicles, pedestrians, and a 3D-LiDAR installed at the target intersection. The traffic is generated via the CARLA interface with respect to certain traffic demands and the traffic signals are controlled by the built-in traffic signal manager in CARLA.

In this paper, we implement a roadside 3D-LiDAR as our main sensor. The roadside LiDAR is installed at the southwest corner of the target intersection, which is demonstrated in Figure 3 (the LiDAR is installed below the arm of the traffic signal pole). Specifically, detailed settings about the roadside 3D-LiDAR are described later in Table I. To reduce the concern on transferability of the deployed deep neural network model, we resemble the LiDAR setups as used to obtain the KITTI dataset [40]. This means that the pitch, yaw, and roll of the LiDAR are set as zeros in CARLA global coordinate, as shown in the right part of Figure 3. Specifically, the LiDAR intensity is calculated by the following equation:

where $I_{0}$ represents the initial intensity value (equals to 1 in this study); $a$ represents the attenuation coefficient, depending on the sensor’s wavelength and atmospheric conditions (which can be modified by the LiDAR attribute “atmosphere_attenuation_rate”); and $d$ is the distance from the hit point to the sensor. Furthermore, realistic LiDAR features such as random no returns and background noise are also considered [41]. More details about the implemented 3D-LiDAR are described in section IV.

In this paper, we apply the ComplexYolo model [37] as our fundamental 3D object detection method. The basic pipeline of the ComplexYolo model is demonstrated in Figure 4.

In the ComplexYolo model, raw 3D-LiDAR data is firstly cut into a certain shape with respect to the target region; then the 3D point cloud is processed into Bird’s-Eye-View (BEV) RGB map based on different features; the CNN-based ComplexYolo network generates detection outputs based on the RGB map; moreover, post-processing is implemented to filter detection results with respect to certain thresholds; finally, 3D bounding boxes are calculated and displayed on the RGB map image. The optimization loss functions $L$ for ComplexYolo is defined as:

where $L_{Yolo}$ is defined as the sum of squared errors using the multi-part loss introduced in YOLO [42] and YOLOv2 [43], while the Euler regression part $L_{Euler}$ is defined to handle complex numbers, which has a closed mathematical space for angle comparisons [37]. The implementation details of the ComplexYolo model in this paper will be introduced in Section IV.

Owing to the co-simulation structure, the communication delay is naturally involved in deploying Realworld Simulation and Mirror Simulation on different computers. Furthermore, to support assessment for different delay conditions, an active communication delay (ACD) is designed at Detection Generator shown in Figure 5. ACD aims to provide customized time delays that can be defined according to specific requirements. Hence, the total delay $\mathcal{T}$ of the co-simulation is designed as:

where $\mathcal{T}_{In}$, and $\mathcal{T}_{ACD}$ represent the innate time delay of the co-simulation and the delay from ACD. Specifically, for wireless communication applied in the our case study shown in Section IV, $\mathcal{T}_{In}$ is about $150ms$ and $\mathcal{T}_{ACD}\sim\mathcal{N}(\mu,\sigma^{2})$ is applied by setting $\mu,\sigma$ as $50$ and $5$ respectively.

Considering that message dropping is a common issue in communications systems [46], an Active Message Dropping (AMD) mechanism is designed in our co-simulation system. In the Co-simulation system, AMD is designed by the stochastic process and deployed right after the ACD module. The $\mathcal{R}$ represents the factor for the message dropping, which is designed by:

For each frame, if $\mathcal{R}$ is smaller than a certain threshold $\eta$, the message of this frame will be drooped. Specifically, for experiments in section IV-E, $\eta$ is set as $0.0$, $0.05$, and $0.10$ respectively.

In this paper, our target range for vehicle detection is defined as a 50 meters by 50 meters area $\Omega$ with respect to the location of LiDAR. The square size of the target range is due to the construction mechanism of the BEV feature map which is shown in section IV-C2. The raw point cloud data can be described by:

To reduce the impact from 3D LiDAR points out of the target range, $\mathcal{P}$ is geo-fenced by:

where $\mathcal{P}_{\Omega}$ represents the 3D point cloud data after geofencing; and $\mathcal{X}$ and $\mathcal{Y}$ are set as $[0.0m,50.0m],[-25.0,25.0]$ respectively. Considering the calibrated height of the roadside Lidar to be $1.74m$, $\mathcal{Z}$ is set as $[-2.74m,1.36m]$. Additionally, ground truth labels are collected according to the objects within the target range.

The 3D points within the target range are then normalized by the following equations:

where $range_{x}$ and $range_{y}$ represent the range along the x-axis direction and y-axis direction, respectively; $w$ and $h$ denote the weight and height of the BEV image, respectively. Specifically, $[x\ y\ z]^{T}$ and $[\tilde{x}\ \tilde{y}\ \tilde{z}]^{T}$ represent the points in $\mathcal{P}_{\Omega}$ and the points after normalization respectively.

Then the three-feature maps, i.e., R-map, G-map, and B-map can be defined according to the density, height, and intensity information, respectively, which is shown below:

where $S_{j}$ represents a specific grid cell of RGB-map; $z_{r}$ , $z_{g}$ and $z_{b}$ represent three channels for RGB-map; $I$ represents the intensity of LiDAR point and $N$ describes the number of points mapped from $P_{\Omega\,\ i}$ to $S_{j}$.

Based on the CARTI dataset, we train the ComplexYolo model from scratch via stochastic gradient descent with a weight decay of 0.0005 and momentum of 0.9. For the dataset preparation, we subdivide the training set with 80% for training and 20% for validation. The learning rate is set as 0.001 for initialization and gradually decreased along 1000 training epochs. For regularization, we implement batch normalization. For activation functions, a leaky rectified linear activation function, defined as follows, is used except in the last layer of the convolution neural network (CNN) where a linear activation function $f(x)=x$ is used.

For quantitative results and analysis, Figure 6 demonstrate the overall performance of the training results including the training loss, precision, and recall. Specifically, Precision illustrates the proportion of true positive detection among all predicted detection (i.e., all the bounding boxes generated from the model). The recall is a factor that measures the ability of the detector to find all the relevant cases (i.e., all the ground truths) which is the proportion of true positive detection among all ground truths (i.e., real vehicles and pedestrians). Additionally, the notations of $@50$ and $@75$ mean these values are calculated based on the Intersection of Union (IoU) of $0.50$ and $0.75$, respectively.

The evaluation results of the testing dataset are demonstrated in Table II. Specifically, AP represents the average precision for each class along the different thresholds and F1 is the harmonic mean for precision and recall, which is designed as:

It is notable that the detection results for vehicles are much better than for pedestrians. A hypothesis is that fewer LiDAR points would be reflected from pedestrians than vehicles and pedestrians are more susceptible to occlusion, due to their smaller sizes.

For qualitative results and analysis, Figure 7 demonstrates the key pipeline for the co-simulation system. The left-column images show the top-down view of the “real-world” traffic environment, the authentic perception results are visualized in mid-column images, and the traffic conditions in the “mirror world” are demonstrated by the right-column top-down view images. For detection visualization, vehicles and pedestrians are bounded with yellow and red boxes, respectively. In addition, the blue edges of the bounding boxes represent the forward direction of the objects.

Figure 7 validates the feasibility of our CMM-based co-simulation platform. Vehicles and pedestrians are detected via the roadside-based LiDAR and the associated digital replica is reconstructed in the mobility mirror simulator. Due to the detection range and accuracy of the selected model (i.e., ComplexYolo), some objects will be missed. Nevertheless, our platform is generic and highly compatible with different detection models and the results can be improved with the advances in SOTA detection methods.

In general, Figure 7 validates the core concept of our CMM-based co-simulation platform, i.e., generating authentic detection results based on high-fidelity sensors and rebuilding the traffic objects for external CDA applications, which demonstrates the functionality and feasibility of the co-simulation platform.