Automatic Map Generation for Autonomous Driving System Testing

Based on [16], lanes can be represented by their center reference paths. We model these reference paths using cubic Bézier curves, which are extensively used in computer graphics [17]. Given four control points $P_{0}$, $P_{1}$, $P_{2}$ and $P_{3}$, the cubic Bézier curves can be formulated as:

where $t\in[0,1]$, and $\binom{3}{i}$ are the binomial coefficients. Cubic Bézier curves are powerful in representing simple curves such as straight lines, circle-shaped or S-shaped, and can be combined into Bézier splines of complex shapes. For any path (e.g. reference path or boundary curve of a lane), if the start point $P_{start}$, start heading $\theta_{0}$, end point $P_{end}$, and end heading $\theta_{1}$ are known, the controls points can be calculated as:

where both $d_{0}$ and $d_{1}$ are heuristically defined as half of the distance between $P_{0}$ and $P_{3}$ for a smoother curve.

In the section, we define the junction features to characterize a junction, containing the road feature, the rotation feature, the control feature, and the crosswalk feature.

First, the road feature aims to describe the number of roads connected by a junction, such as T-junctions and n-legged intersections. The more connected roads, the more complex the junction becomes, resulting in more testing efforts for ADSs. Hence, the road feature of a junction $J_{j}$, denoted as $F_{J_{j}}^{road}$, is defined as the number of roads connected by a junction. For example, the junction $J_{0}$ shown in Fig. 1 connects four roads, so $F_{J_{0}}^{road}=4$, while $F_{J_{1}}^{road}=3$.

Second, the rotation feature aims to model the relative orientations of the roads connected by a junction. It indicates the relative directions an autonomous vehicle can drive after entering the junction. Given a junction $J_{j}$, suppose its center is $C$, which is calculated as the geometry center of its bounding polygon, and the corresponding start point of road $R_{i}$ is $P_{i}$, which is the connecting endpoint of $R_{i}$’s reference line to the junction. The vector $\vv{CP_{i}}$ is called road socket of $R_{i}$. Hence, the relative location of $R_{i}$ and $R_{k}$ can be measured by the angle between $\vv{CP_{i}}$ and $\vv{CP_{k}}$. Hence, the rotation feature is a vector of angles, each describing the angle from one road socket to its adjacent one in counter-clockwise order. Given a junction $F_{j}$, its rotation feature is denoted as $F_{J_{j}}^{rot}=[\gamma_{1},\gamma_{2},\ldots,\gamma_{N}]$, where $N=F_{J_{j}}^{road}$ and $\sum_{i=0}^{N}\gamma_{i}=2\pi$.

Given such a rotation feature, we can generate various junctions by rotating the original junction around its center. Hence, we may first perform rotation normalization, i.e., rotating each junction such that it aligns to the four regular directions, i.e., East, North, West, and South, as much as possible. The rotation normalization process can be described as follows. Suppose the rotation feature of a junction $J_{j}$ is $[\gamma_{1},\gamma_{2},\ldots,\gamma_{N}]$, where $N=F_{J_{j}}^{road}$. We first rotate and re-index the road sockets and the angles such that $\gamma_{1}$ is the maximal one. After rotation normalization, the angle from the East direction to $CP_{1}$ is $\alpha_{opt}$. Then, $\alpha_{opt}$ can be the optimal solution by solving the following optimization problem:

were $\beta_{i,1}$ (resp., $\beta_{i,2}$, $\beta_{i,3}$, and $\beta_{i,4}$) is the angle from East (resp., North, West, South) to $\vv{CP_{i}}$. Thus, the rotation feature can be represented by the angles from the East direction to the road sockets, i.e., $F_{J_{j}}^{rot}=[\alpha_{1},\alpha_{2},\ldots,\alpha_{N}]$, where $\alpha_{i}=\beta_{i,1}$ is computed by Eqns. 6 and 7 with $\alpha=\alpha^{opt}$. With the normalized rotation feature, the corresponding road sockets are uniquely determined. For example, as shown in Fig. 2, suppose after rotation normalization, $\alpha^{opt}=\alpha_{1}$, then we have the rotation feature $F_{J_{j}}^{rot}=[\alpha_{1},\alpha_{1}+\gamma_{1},\alpha_{1}+\gamma_{1}+\gamma_{2}]$.

Third, the control feature designates how the traffic is regulated in a junction. Commonly, there exist three kinds of control manners, i.e., $bare$: no controls, $signal$: controlled by traffic lights, and $stop$: controlled by stop signs. Hence, the control feature can be defined as $F_{J_{j}}^{ctrl}\in\{bare,signal,stop\}$. For example, in Fig. 1, $F_{J_{0}}^{ctrl}=signal$ and $F_{J_{1}}^{ctrl}=stop$.

Finally, the crosswalk feature, denoted as $F_{J_{j}}^{xwlk}$, indicates whether there are crosswalks in a junction. Hence, $F_{J_{j}}^{xwlk}\in\{True,False\}$.

Therefore, we have the following definition.

According to Definition 1, we describe how to extract the features from a set of HD maps.

First, since the road, the control, and the crosswalk features are discrete variables, we can obtain all their values by checking the junctions in each map. Therefore, the sets of all possible values of the three features of a given map are denoted as $\mathcal{F}^{road}$, $\mathcal{F}^{ctrl}$, and $\mathcal{F}^{xwlk}$, respectively.

Second, as the rotation feature contains continuous variables, we collect from the HD maps the minimal and the maximal values of each element in the rotation feature. Note that the rotation features from the junctions with the same road feature can be compared element-wisely after rotation normalization. Hence, the collected rotation feature for the junctions with $n$ connected roads can be described as:

where $\alpha_{i}^{\min}=\min\limits_{J_{j}\in\mathcal{J}^{n}}F_{J_{j}}^{rot}[i]$, $\alpha_{i}^{\max}=\max\limits_{J_{j}\in\mathcal{J}^{n}}F_{J_{j}}^{rot}[i]$, $\mathcal{J}^{n}=\{J_{j}:F_{J_{j}}^{road}=n\}$, and $F_{J_{j}}^{rot}[i]=\alpha_{i}$ with $F_{J_{j}}^{rot}=[\alpha_{1},\ldots,\alpha_{n}]$.

Hence, the map feature of a set of maps is described as $\mathcal{F}=\bigcup_{n\in\mathcal{F}^{road}}\mathcal{F}_{n}^{rot}\times\mathcal{F}^{ctrl}\times\mathcal{F}^{xwlk}$.

During the map construction with the junctions generated by the feature vectors, we need to ensure 1) all junctions are connected, 2) the generated map emulates urban road networks, and 3) no road or junction overlaps, which makes the construction process challenging. In this paper, we propose a greedy method to construct a grid-layout map iteratively. The details are shown in Algorithm 1.

Given a set of HD maps $\mathcal{M}$, by checking all junctions of the maps, we can extract the features $\mathcal{F}^{road}$, $\{\mathcal{F}_{n}^{rot}:n\in\mathcal{F}^{road}\}$, $\mathcal{F}^{ctrl}$, and $\mathcal{F}^{xwlk}$. Before constructing the map, we initialize a grid point container $\mathcal{P}_{e}=\{P_{0,0}\}$ for holding empty points, another grid point container $\mathcal{P}_{f}=\emptyset$ for filled points and a set $\mathcal{J}=\emptyset$ for storing sampled junction features (Line 1). For each vector $[F^{road}_{J_{j}},F_{J_{j}}^{ctrl},F_{J_{j}}^{xwlk}]$ in $\mathcal{F}^{road}\times\mathcal{F}^{ctrl}\times\mathcal{F}^{xwlk}$, we perform multivariate uniform sampling in the region $\prod_{i=1}^{n}[\mathcal{F}_{n}^{rot}[i][0],\mathcal{F}_{n}^{rot}[i][1]]$ to generate a junction feature $\mathcal{F}_{J_{j}}=[F^{road}_{J_{j}},F_{J_{j}}^{rot},F_{J_{j}}^{ctrl},F_{J_{j}}^{xwlk}]$ (Lines 1 - 1). The sampled junction features are stored in $\mathcal{J}$ (Line 1), each of which will be used to instantiate an individual junction in the grid layout.

Next, for each junction feature $\mathcal{J}_{j}$, we search for the empty grid points in $\mathcal{P}_{e}$ to select the grid point $P_{x,y}$ of the highest matching score with $\mathcal{J}_{j}$. The matching score is heuristically calculated in the following way:

Let $k$ be the maximal number of roads that $J_{j}$ at $P_{x,y}$ can connect after proper rotation with the surrounding junctions (if available) at $P_{x+1,y}$, $P_{x,y+1}$, $P_{x-1,y}$, and $P_{x,y-1}$. If $F^{road}_{J_{j}}>k$, then the feature and the point does not match and the score is set to $-\infty$. Otherwise, the score is set to $k$. Random selection is performed when a draw occurs.

After the grid point, say $P_{x,y}$, is selected for $\mathcal{F}_{J_{j}}$, and the corresponding junction is instantiated at $P_{x,y}$ (Line 1-1). The junction feature is removed from the feature set (Line 1) and the grid is extended in the four regular directions (i.e., East, North, West, and South) from $P_{x,y}$, free for new junction assignments (Line 1-1). After all the junctions are instantiated, the roads connecting junctions at neighboring grid points can be constructed via road socket vectors (controlled by $F^{rot}$). The start point (resp. end point) of road lanes can then be obtained by shifting the start point (resp. end point) of the road’s reference path along the perpendicular direction of the start heading (resp. end heading) of the road. After the road lanes are built, the junction lanes can be added by connecting the incoming and outgoing lanes of the junction. Next, traffic controls and crosswalks are created at appropriate locations. Lastly, we can output the HD map files based on $\mathcal{M}_{c}$.

Note that to make the generated HD map as concise as possible while preserving potential scenario diversity, we set the following default configurations for Feat2Map:

• •

  Given two junctions with the same junction feature, but one contains a one-way road while the other contains only two-way roads, any testing scenario generated from the former can also be generated from the latter one. Hence, all roads are two-way to keep the generated map concise.

• •

  To minimize the number of junctions in the generated map, we assume that all roads are fully connected at junctions. It means there exists at least one navigation path (i.e., junction lane) connecting each road pair.

• •

  The crosswalks are instantiated within the range of junctions and only overlap with junction lanes instead of road lanes. The signals are placed in front of the road it controls (facing the road), and the stop signs are placed on the right side of the road (facing the road).

• •

  The default grid-gap is 100 meters (e.g., $P_{x+1,y}$ is 100 meters to the east of $P_{x,y}$, lane width is 3.5 meters (default value of selected input maps), and crosswalk width is 4.5 meters (slightly longer than a regular sedan).

To demonstrate the effectiveness of Feat2Map, we implement a prototype of Feat2Map and conduct experiments on the latest Apollo 7.0 [14] stack with three of the largest HD maps available for Apollo, i.e., $SanFrancisco$ [18], $GoMentum$ [19] and $Shalun$ [20]. We define two metrics to measure scenario diversity. The first one is line coverage, which describes how many statements in the ADS have been executed by the scenarios. Two scenarios are considered different if they cover different statements. The other one is behavior coverage, which describes different behavior transitions during the execution of a scenario.

To answer RQ1, we run Feat2Map on $SanFrancisco$ and perform qualitative analysis on three generated maps $M_{1-3}$. To answer RQ2, we conduct quantitative analysis by comparing the coverage results on $SanFrancisco$ and $M_{1-3}$. To answer RQ3, we generate a new map $M_{4}$ from $SanFrancisco$ [18], $GoMentum$ [19] and $Shalun$ [20] and conduct the same experiments as in RQ2. To answer RQ4, we illustrate the construction of customized maps from manual features.

All the experiments are conducted on a desktop computer (with i7-6850K CPU, 64GB RAM, and GTX 1080T Ti). When an HD map is generated, Dreamview (Apollo’s full-stack HMI service) [21] is restarted to load the new map. We run the route coverage testing [4] in Apollo’s built-in sim-control mode [22], which does not require any third-party simulators such as SVL [10]. The input of the planning component is published using simulated dummy data. Specifically, sim-control publishes the results of $localization$, $chassis$, and $prediction$, while our testing script submits $routing\ requests$ and publishes $traffic\ light$ information at runtime.

San Francisco map contains 91 junctions (including 88 signal-controlled and three stop-sign controlled), 164 roads, and 1524 lanes (including 868 junction lanes and 656 road lanes); there are 13 junctions with crosswalks.

Due to randomness in the rotation sampling of road socket vectors, the extracted feature set $\mathcal{F}$ can generate unlimited HD maps. Fig. 3 presents the bird-view of the input map $SanFrancisco$ and three generated HD maps $M_{1}$, $M_{2}$ and $M_{3}$ based on extracted features listed in Table I. We can find that the number of junctions is reduced to 8 in $M_{1-3}$ from 91 in $SanFrancisco$.

Take $M_{1}$ in Fig. 3(b) for example. According to Algorithm 1 given in Section III-D, Feat2Map generates 8 junctions, as shown in Fig. 4. Junction $J_{0}$, $J_{4}$, $J_{5}$, and $J_{7}$ are three-legged T-junctions and the rest are four-legged intersections. Junction $J_{0}$, $J_{3}$, $J_{5}$, and $J_{6}$ are signal-controlled while the rest are stop-sign controlled. Junction $J_{2}$, $J_{5}$, $J_{6}$, and $J_{7}$ have crosswalks instantiated at all the connected roads while the rest have no crosswalks. From Fig. 4, we can see that Feat2Map can generate junctions with different topological and geometrical features and instantiate traffic controls and crosswalks correctly.

To collect the line coverage, we instrument Apollo’s planning component during compilation using the coverage command [23] provided by its build tool Bazel [24] as the scenario handling mechanisms are implemented in the planning component. Fig. 5 shows the coverage results of our experiments on the four maps in Fig. 3. Fig. 5(a) shows the accumulated code coverage results on different maps. The accumulated lines covered eventually are 8275 ($SanFrancisco$), 8232 ($M_{1}$), 8205 ($M_{2}$) and 8238 ($M_{3}$) respectively. As we can see, $SanFrancisco$ has many junctions covering the same lines of code (highlighted in light blue), resulting in a significant amount of duplicated test cases. On the other hand, unique lines of code get covered in every junction in $M_{1-3}$. It only takes eight junctions in $M_{1-3}$ to reach a similar level of code coverage. Notice that $SanFrancisco$ has slightly higher code coverage than $M_{1-3}$. Based on the source code-level investigation, the uniquely covered codes in $SanFrancisco$ are related to human errors in the HD map file (e.g., the id of some crosswalk elements are missing). Such human errors do not exist in maps generated by Feat2Map automatically. Besides the map error, we also discovered some potential issues of the planning component (reported at [25, 26]). Such issues, encountered in different situations (e.g., on roads or inside junctions), trigger different error handling mechanisms implemented in the planning component. This is the main reason for the slight difference in the line coverage results of $M_{1-3}$.

Apollo implements its planning component as a modularized state machine, where states align with common traffic scenarios such as intersection cruising, emergency stops, etc. and publishes its internal scenario selections (e.g., LaneFollow and TrafficLightProtected) and stage transitions (e.g. Approach and IntersectionCruise) within each scenario.

The coverage results of the stage transitions are shown in Fig. 5(b). From the results, we can find that many junctions belong to the same scenario in Apollo’s view of point. As a result, the same strategies are applied by Apollo to navigate through them (highlighted in light blue as well). On the other hand, the eight junctions in $M_{1-3}$ reach the same level of coverage as that of 91 junctions in the $SanFrancisco$ map. Specifically, there are 16 scenario types defined in Apollo [27], out of which five scenarios and their stages are relevant to the experiments [28]:

• •

  Lane Follow

• •

  Stop Sign Unprotected

• •

  Traffic Light Protected

• •

  Traffic Light Unprotected Left Turn

• •

  Traffic Light Unprotected Right Turn

The relevant scenarios and their stage transitions are plotted in Fig. 6, where the nodes represent the stages and arrows are possible transitions. The stages and transitions covered by $SanFrancisco$ and $M_{1-3}$ are the same and highlighted in orange, while the uncovered stages are rendered grey.

The four paths covered are:

• •

  $LF\rightarrow S\_PS\rightarrow S\_S\rightarrow S\_C\rightarrow S\_I\rightarrow LF$

• •

  $LF\rightarrow T\_A\rightarrow T\_C\rightarrow LF$

• •

  $LF\rightarrow L\_A\rightarrow L\_I\rightarrow LF$

• •

  $LF\rightarrow L\_A\rightarrow L\_C\rightarrow L\_I\rightarrow LF$

The stage transitions align with our testing setup, starting and ending at road lanes (i.e., $LF$). The covered scenarios also match with the features of the generated junctions. One may notice that the right turn-related scenarios are not covered. This is because Apollo only enters the right turn scenario when the ADS is turning right under the red traffic light. When the traffic light is green, which is always the case during our $sim-control$ simulation, Apollo treats right turning the same as driving straight across the junction (i.e., $T\_A$ and $T\_C$).

We generate map $M_{4}$ with multiple input maps, i.e., $SanFrancisco$, $GoMentum$, and $Shalun$. The experiment results are listed in Table II. The features extracted from $Shalun$ are not much different from $SanFrancisco$. However, due to the existence of the bare junctions (i.e., no controls) in $GoMentum$, the merged control feature is $\mathcal{F}^{ctrl}=\{signal,stop,bare\}$, which results in 12 junctions to be generated in $M_{4}$. The code coverage of $M_{4}$ is similar to those of $M_{1-3}$. We find that Apollo 7.0 treats bare junctions as normal roads, hence no significant difference in the code coverage. However, a new path consisting of only one stage $LF$ (i.e., no junction scenarios are triggered) is covered because of the existence of bare junctions.

In summary, experiments with single and multiple input maps have demonstrated the effectiveness of Feat2Map in feature-based map generation.

The generation capability of Feat2Map is not limited to generating junctions from the existing maps, such as those shown in Fig. 4. Feat2Map also accepts manually defined feature values if users require customization. For example, the mega junction shown in Fig. 7(a) is generated by specifying the following junction feature: $F_{mega}^{road}=5$, $F_{mega}^{ctrl}=signal$, $F_{mega}^{xwlk}=True$ and $F_{mega}^{rot}=[0^{\circ},72^{\circ},144^{\circ},-144^{\circ},-72^{\circ}]$.

With the cubic Bézier curve model, Feat2Map can also generate a set of roads connected end-to-end by specifying a list of road configurations where each road’s end point (resp. end heading) is the next road’s start point (resp. start heading). Finally, in addition to the default two-way roads, one-way roads are also supported by Feat2Map as long as users specify the road type of each road socket. Fig. 7(c) shows a T-junction where the road types associated to the sequence of the road sockets, started from the East most road socket and listed in a counterclockwise direction, is $[In,Out,InOut]$, where $In$, $Out$, $InOut$ mean one-way incoming, one-way outgoing, and two-way roads, respectively.