DeepCrashTest: Turning Dashcam Videos into Virtual Crash Tests for Automated Driving Systems

For automated vehicle tracking, we use a combination of object detector and tracker along with standard association algorithms since object trackers need annotations of initial bounding boxes for the vehicles. There are many well known fast object detection networks such as [21, 22, 23, 24] but we chose Mask-RCNN [25] which is an instance segmentation method which simultaneously detects objects and estimates segmented masks. Region proposal based networks like Mask-RCNN are flexible with input image dimensions which is an important criterion for our choice. Moreover, instance segmentation is necessary for an alternate approach to extract 3D bounding box centers in sec. IV-C.

For object tracking, traditional trackers [26, 27] use Kalman filters which need proper camera calibrations and fine-tuning of the motion or sensor model is generally difficult. We use the deep-learning based Re3-Tracker [28] which is robust to occlusion and has real-time capabilities.

For data association between object detector and tracker to add or remove excess trackers in a frame, we use the Hungarian algorithm [29] with Intersection Over Union (IOU) of bounding boxes as the matching criteria. As in [30], we define threshold parameters to determine the minimum number of consecutive frames with consistent predictions for removing or adding excess trackers.

We can approximate a vehicle’s trajectory (position and heading) by estimating it’s sequence of 3D bounding box coordinates at each frame from the video. Using the frame rate of video, we can approximate the relative velocities of the agents with respect to ego at all time instants. We remark that there is a high possibility of variation in the original frame rate during processing and uploading of video to internet which does not guarantee the accuracy of absolute velocity estimates. We can however safely assume that the relative trend in velocity profiles between vehicles remains unaffected. This is found to be sufficient to ensure the timing of vehicle movements and recreate the crash scenario.

Over methods like [31, 32, 33, 34], we chose 3D-Deepbox network [35] because of its simplicity and good generalization of the approach to random images. They use a VGG-Net backend with additional fully connected branches to extract the bound box dimensions along with the global orientation of the vehicles. The 3D center of the bounding box with respect to camera is further optimized by placing geometric constraints on the images.

Camera calibration is required at this stage to extract real world distances. We observed that choosing a camera intrinsic close to the dataset which the network is initially trained on (KITTI Dataset [36] in this case) gives a satisfactory output on quite a number of random images. We review an existing method to approximate camera parameters from lane lines in sec. IV-F. The extracted trajectories are sent into the second stage of framework for simulation.

In the previous method for 3D center estimation, we have directly utilized the monocular 3D object detection networks. There is an alternative approach using depth maps of a scene for extracting only the 3D positions or specifically the distance and lateral position of the vehicle in a calibrated sensor environment with good quality images. Given a depth map, the 3D point cloud can be generated by estimating the real world $(x,y,z)$ coordinates of individual pixels $(u,v)$ considering the depth D of scene to be in the z direction using notation and equations as in [37] (see Fig. 3):

where $(c_{u},c_{v})$ is the principal point and $(f_{u},f_{v})$ is the horizontal and vertical focal length of the camera in pixel coordinates. We assume the principal point to be the center of the image with equal horizontal and vertical focal length.

From the existing CNN architectures for monocular depth extraction [38, 39, 40], we use Monodepth [38] because of its longer range for accurate depth prediction. Initial instance segmentations from the mask-RCNN network are used to crop the 3D point cloud corresponding to a specific vehicle. We simply find the mean of these 3D points to approximate the vehicle’s position. An important clarification to be made is that, we do not consider the 3D coordinates estimated using this approach to be the actual center of the vehicle, but just an approximate position in the scene. This is due to the fact that extracted point clouds belong only to the part of the vehicle which is exposed to the camera. However, the error in Euclidean distance between actual center and the estimated position does not exceed the dimensions of vehicle. Hence, the path formed by this sequence of 3D points at all frames provides information about the vehicle trajectory.

Other works on similar lines are [37, 41, 42]. Their methods are much more refined and accurate since they use additional network architectures for processing. We are using a simple method which is sufficient for extracting approximate positions. There are even end-to-end networks [43, 44] for joint monocular depth extraction and ego-motion estimation but we prefer the modular approach due to it’s good generalization capabilities.

For the simulations, we need the ego vehicle’s motion or camera motion to find the absolute trajectories of the agent vehicles with respect to the world frame. In case of simple highway scenarios with the ego moving in a straight line, we can assign constant velocity and a linear path for the ego vehicle. To generalize well to arbitrary scenarios, we resorted to monocular Simultaneous Localization and Mapping (SLAM), Visual Odometry (VO), and Optical Flow based approaches to estimate the ego motion.

There are numerous works on Visual SLAM and VO and comparisons are provided by [45, 46, 47]. We evaluated several methods like DSO [48], LSD-SLAM [49], ORB-SLAM [50] and even Deep learning based approaches like Deep-VO [51]. We chose the former model based approaches because of their lightweight real time capabilities.

However, we found them to introduce scale ambiguity and, also, they cannot provide real-world depth estimates which further deteriorates velocity estimates. There are a few elegant approaches built on optical flow methods which can provide good estimates with lower scale ambiguities. Florian Raudies et al. [52] present a survey of optical flow based ego-motion and depth estimation methods. We use the work by Andrew Jaegle et al.[53] where they introduce Expected Residual Likelihood (ERL) to remove outliers in optical flow estimates and a novel formulation of camera motion equation using lifted kernel optimization framework.

We observe that any of the above methods have stable values in the longitudinal direction of the motion of the vehicle, but they suffer from lateral drifts in the ego position. Currently, we are using lane detection and tracking methods (see next section) to localize the vehicles within lanes and prevent too much lateral drift on the road. Lane detection and tracking helps in camera calibration as well.

The Ego-motion estimator returns a qualitatively correct trajectory which may not be accurate enough to laterally localize the ego vehicle in the correct lane. The lane change information can even be used to completely replace the lateral ego-motion estimates in case of high drifting predictions.

Standard algorithms like the CHEVP [54] or the latest ones like [55] could be used for lane detection, but they fail when there are too many occlusions on the lane lines in high traffic scenarios and do not work well with partial lane markings. We use the deep learning based LaneNet [56] network which is a segmentation based approach comparatively more robust to partial or occluded lane markings. We use a density based spatial clustering algorithm (DBSCAN) along with B-spline curve fitting with the segmented outputs of LaneNet to cluster and assign individual lane IDs.

For lane tracking, there are existing methods (e.g, [57, 58]) involving kalman filters, but we use a simple approach where we fit straight lines through the clustered lane pixels and the lane fitting is completely in image pixel domain. We only need information about the lower part of the lane lines which is close to the ego vehicle to localize within the lanes. The Hungarian algorithm with directed Hausdorff distance [59] as a cost metric is used for association between detected and tracked lanes at each frame.

Considering the bottom left corner of the image as the origin, we find the x-intercepts of the lane lines in pixel coordinates. Since the perspective effects or distortions in an image increase only with depth of the scene, we can safely consider that the distance between any of the x-intercepts in image space is proportional to the actual distance between the corresponding lanes. Since, the average lane width on a specific road segment remains fairly constant, we can estimate the lateral position of the vehicle within the lanes.

Camera calibration is required by the 3D bounding box detection, depth extraction network and optical flow based ego-motion estimation. Due to the monocular camera setting, it is very challenging to form a closed form solution using the image features. One approach is to use learning methods like [60][61] trained on existing datasets [62] to estimate the focal length given an RGB image.Fung et al. [63] demonstrate a more simple and light-weight approach which is the reformulation of perspective camera equations and using the lane line annotations to estimate the camera parameters. We use this method only when the predicted lane lines are pretty accurate and in the case of errors, we use the parameters from the datasets used for training the networks. We decide this based on the visual correctness of the final 3D bounding boxes when plotted on the video frames.

We use a Savitzky-Golay filter followed by a spline smoothing for the raw trajectories. Two levels of spline smoothing is used, first is local windowed with high smoothness factor and, then, a global fitting with low smoothing factor. This ensures an overall smooth curve while still preserving the sudden abnormal vehicle movements that lead to collisions. Moreover, splines provide the needed trajectory parameterization to use test generation frameworks such as sim-ATAV [7]. The extracted lateral localization from lane predictions is added to the corresponding ego position at each frame in this stage.

The agent vehicle trajectories can be categorized based on their heading direction and the time of appearance or disappearance from the scene. Agent vehicles can either move in same or opposite direction of ego i.e, ongoing or oncoming, respectively.

The ongoing vehicles can be classified into three types:
D0T1: Agent vehicle is in the scene since the first frame.
D0T2: Agent vehicle enters later into the scene from behind i.e, agent overtaking the ego.
D0T3: Agent vehicle enters later from front in the scene i.e, a far off vehicle slowing down or pulling over or stopped by the roadside. This can even be considered as a case where the ego overtakes an agent vehicle which is initially far ahead from the ego.

The oncoming vehicles can be classified into four types:
D1T1: Agent vehicle already in scene since the first frame and the ego vehicle eventually passes it before the completion of the video sequence.
D1T2: Agent vehicle is in scene since the first frame and the ego vehicle doesnot pass it through the entire video sequence. This may happen when the agent vehicle collides with ego or the video clip ends before the agent reaches the ego.
D1T3: Agent vehicle enters later into the scene and the ego passes it.
D1T4: Agent vehicle enters later into the scene and the ego doesnot pass it.

This kind of classification is necessary for appropriate trajectory processing and is made purely based on our observations of the dashcam crash videos.
Processing Ongoing vehicle trajectories: D0T1 paths can be directly used without any processing, but the D0T2 paths need extrapolation of path and velocity profiles. Delay needs to be added to D0T3 vehicles such that the frame and position at which they appear in the scene with respect to ego is properly synchronized with the original video. For D0T2 vehicles, until the frame where they appear in the scene, their initial positions and velocity profile are extrapolated such that the agent mimics the ego motion. Sufficient distance is maintained such that it does not collide with ego or does not appear in the scene. D0T3 vehicles are initially far ahead of ego and appear much later in the video. To recreate this, we initialize agents at their initial position and then introduce some delay in the controller calculated based on frame of appearance and frame rate of video. In all the ongoing vehicle trajectories, if the ego overtakes other vehicles and the agent leaves scene before end of video, we extrapolate these trajectories until the end of the simulation following the road or lane at a velocity less than the ego.
Processing oncoming vehicle trajectories: The D1T1 and D1T3 vehicles are passed by the ego in the video and we do not have information once they move out of the scene. We extrapolate these trajectories such that they follow the road with a constant velocity after they pass the ego. D1T3 and D1T4 enter later into the scene and similar to D0T3 vehicles, we generate initial start delays.

Generating proper road structure is important for recreating plausible simulation. In most of the cases, since the crash happens on an highway, we can simply generate a straight road structure. However, it is also important to reproduce curved roads as observed in one of our video demonstrations which involves an oncoming collision on a sharp turning. We use the ego vehicle’s qualitative trajectory as reference to generate the road structure and perform spline smoothing with a high smoothness factor. For oncoming collisions, the ego trajectory does not have any information about the road structure beyond the point of collision. However, it is necessary to extend the road beyond the collision point. One such example can be observed in the head on collision simulation in our video submission. Similar to the road structure generated using ego vehicle paths, we use the oncoming agent paths to extend the road by using spline interpolation to generate a continuous road structure. The entire process is automated in the framework. This approach does not guarantee accurate road estimation but is found to generate qualitatively correct simulations.

Typically, in the available video clips, the vehicles move with non zero velocity from the initial frame. Since starting vehicles with non-zero velocities in Webots simulations does not result in convincing natural movements, we need the vehicles from zero initial velocity until all the vehicles reach the intended velocities and relative positions at the right time.

The time taken or distance traveled in a straight path to reach that target velocity under constant acceleration can be directly calculated from kinematic equations. We will refer to this as step-back time and step-back distance, respectively. Since the target velocities of all the vehicles where they first appear in the scene are different, each vehicle needs varying step-back times and distances. Additionally, we need to synchronize the relative positions of all the vehicles with those estimated in the initial frame of the video by the time they reach their respective target velocities.

To ensure this, we first determine the maximum of all step-back times $t^{s}_{max}=\max{(t^{s}_{1},t^{s}_{2},...,t^{s}_{n})}$, compute the total step-back distance $D^{s}_{i}$ for each vehicle $i$ with step-back time $t^{s}_{i}$ as $D^{s}_{i}$ = $d^{s}_{i}+(t^{s}_{max}-t^{s}_{i})*v^{t}_{i}$, where $d^{s}_{i}$ is the step-back distance for vehicle $i$ to reach its target velocity $v^{t}_{i}$, estimated for its initial frame of appearance and $n$ is the number of vehicles of interest to be simulated. We use the calculated distances as the length for extending the initial segment of the paths. The vehicles follow a straight path until they merge with their actual velocity and path profiles in the video. The part of the simulation which represents or recreates the actual video starts after time $t^{s}_{max}$.

Additionally, we provide visualization and editing tools for minor modifications to the lateral or longitudinal displacements of individual trajectory waypoints either to easily change the initial conditions or to ensure that vehicles do not overlap during initialization. We reiterate that since our target application is testing in a virtual environment which is going to be further modified, we do not need a quantitatively accurate reproduction of the real driving scenario. In other words, a visual inspection for qualitative correctness is sufficient for our application.

In this section, we describe our procedure for a qualitative assessment of our framework through a typical usage of the extracted adversarial trajectories in a test environment for a collision avoidance system. We use a naive Automatic Emergency Braking (AEB) controller from the Sim-ATAV [7] framework. The AEB in sim-ATAV uses a sensor fusion module to perceive the environment by utilizing the sensors attached to the ego vehicle. In our tests, we activate the GPS/IMU, monocular dashcam and a single Radar sensor attached to the front of the ego vehicle to localize the agents in the scene.

Figure 5 shows three time ordered images of an original crash video. Fig. 5 includes the corresponding frames in a simulation generated from the trajectories extracted using DeepCrashTest on the original video. The second and third images in Fig. 5 show that though the AEB activates before the collision, it collides with the agent. This is because, the velocity of ego vehicle is very high (around 90 Kmph) and the distance to obstacle is very low compared to the required braking distance at such high velocities. At these velocities only way to completely avoid a collision is to steer away from the obstacle. To extract safe maneuvers of the ego vehicle which avoids a collision, we generated 128 simulations. For these simulations, we first manually sampled 32 initial positions within a box of 4x8(m) around the original position of ego. For each of these initial positions, we added small 2D guassian noise to the spline control points of the original ego vehicle trajectory to 4 random trajectories. Eight of the 128 simulations did not have a collision. We picked four meaningful ego motions and Fig. 5 includes frames sampled from one of the safe simulation. As can be seen from the third image of 5, before collision, the ego steers away from the agent thus avoiding collision.

Even though, our application focuses on qualitative performance, we provide quantitative evaluations for the trajectory extraction module (see fig. 2). This is not straightforward since, DeepCrashTest deals with an unknown dashcam video without ground truth, however, we provide evaluations by testing the framework with the KITTI tracking dataset [36]. The KITTI benchmark provides tracking evaluation where the object positions are all relative to the ego vehicle but we want to evaluate the absolute trajectory positions by including the ego motion. For this, we added the ground truth odometry information to the ground truth tracking data to evaluate our framework. Table I provides the tracking evaluation results on vehicles using such a modified version for three tracking sequences from the dataset (sequences 3, 8, 10). The actual performance can be slightly lower because of errors in camera calibrations.