RoadPainter: Points Are Ideal Navigators for Topology transformER

The first approach to understanding road structure through lane topology, as described in the study by STSU [1], focused primarily on centerline detection. The method establishes a three-stage paradigm consisting of BEV feature construction, centerline detection, and connection. This paradigm has been widely adopted in most topology reasoning papers. Subsequently, TPLR [2] enhanced the performance by incorporating the concept of the minimum circle formed by the directed centerline segment. The path-wise representation of the centerline, as proposed in LaneGAP [15], effectively maintains the continuity and shape accuracy of the centerline. In order to ensure compliance with traffic regulations at intersections, TopoNet [13] employs Graph Neural Networks (GNN) to establish meaningful connections between driving lanes and traffic signs. By incorporating spatial position encoding, TopoMLP [36] improves the performance of lane relation topology. To further enhance the centerline detection accuracy, the study conducted by SMERF [22] investigates the utilization of SD map. Insights from 3D object detection, such as leveraging the relationship between objects and the driving lane, can enhance the quality of lane centerline extraction, as effectively demonstrated by the OLC method [3].

The field of online map construction is highly relevant to lane topological reasoning. Recent works aim to construct local High Definition (HD) map within a predefined range from onboard sensors, incorporating map elements like lane lines, pedestrian crossings, and curbs. HDMapNet [12] employs semantic learning on BEV features to extract map elements, which are then vectorized through post-processing. Recent advancements in end-to-end learning are being used to eliminate post-processing steps. VectorMapNet [19] utilizes a methodology that identifies coarse boxes of map elements and then generates fine-grained geometric details. InstaGram [29] employs a unique approach that involves initially identifying the geometric vertices of map elements, and then leveraging GNN to learn the associations between these vertices. MapTR [16] and its improved version  [17] decouple the dependency between point association and positioning by assigning a fixed number of points to each map element. An alternative approach called BeMapNet [25] utilizes a unified piecewise Bezier curve method that dynamically adjusts the number of points within a certain range. Pivotnet [6] introduces dynamic matching, retaining only points with relevant geometric features of the map element. ScalableMap [37] enhances long-range perceptual capability by leveraging BEV features guided by the linear structure of map elements.

We also include the subject of lane detection due to its similarities with the aforementioned topics. The early learning-based approaches, including LaneNet [35], UFAST [26], CondLaneNet [18], and GANet [33], extract lane lines from 2D images. LaneNet [35] formulates lane detection as an instance segmentation problem and employs two embedding branches. UFAST [26] adopts feature aggregation and constructs segmentation on a smaller feature map to enhance detection speed. CondLaneNet [18] utilizes shape prior and predicts the line location for each row in the image space, aiming to resolve the instance-level discrimination problem. GANet [33] reformulates the lane detection problem by directly regressing each keypoint to the starting point of the lane line. To achieve more precise lane line detection, researchers have also explored lane detection in 3D space. 3D-LaneNet [7] introduces an architecture that generates 3D lane lines using two pathways: the image-view pathway encodes features in 2D images, while the top-view pathway offers translation-invariant features for 3D lane detection. PersFormer [4] adopts a unified 2D and 3D lane detection framework to simultaneously detect 2D and 3D lane lines. BEV-LaneDet [34] proposes the use of a virtual camera to unify the intrinsic and extrinsic camera parameters, thereby enhancing the performance.

Applying BEV features extracted from online sensors have demonstrated potential for topology reasoning in computer vision tasks. Nevertheless, the presence of challenges such as occlusion and limited sensing range can result in inaccuracies. To address these issues, we present a novel SD map interaction module that effectively augments BEV features by incorporating beyond-visual-range data and road shape priors extracted from the SD map. A piece of an SD map consists of multiple instances denoted as $s_{1},s_{2},\dots,s_{N_{S}}$, where $N_{S}$ represents the total number of SD map instances. Each instance $s_{i}$ is composed of a polyline represented by $\mathbf{sp}_{i}\in\mathbb{R}^{M_{i}\times 3}$, consisting of $M_{i}$ points, and a corresponding semantic type denoted as $st_{i}\in{1,2,\dots,N_{M}}$. Here, $N_{M}$ denotes the total number of semantic types available in the SD map. We begin by converting the vectorized SD map instances into the BEV features. Each grid cell is filled with the corresponding semantic type embedding if it is occupied. Otherwise, a default semantic type embedding is used. This is represented as $\mathbf{E}_{S}\in\mathbb{R}^{HW\times C}$, where $HW$ is the grid dimensions and $C$ is the number of embedding dimensions. The augmented BEV feature map, denoted as $\hat{\mathbf{B}}\in\mathbb{R}^{HW\times C}$, is obtained using the following equation:

where $\mathbf{B}$ represents the BEV features obtained from online sensors, $\mathbf{E}_{P}\in\mathbb{R}^{HW\times C}$ is a positional encoding map for the SD grid, generated using sine and cosine functions of different frequencies [31]. The function TrDec refers to a transformer decoder that incorporates multi-layer deformable self- and cross-attentions among vanilla BEV feature map $\mathbf{B}$ and SD BEV feature map $\mathbf{E}_{S}+\mathbf{E}_{P}$.

The centerline instance points obtained from Section 3.2 often lack sufficient accuracy, especially in cases with high curvature, as shown by those blue dots in Fig. 1. To address this issue, we propose a points-mask optimization module for the refinement of centerline instance points, consisting of two submodules: points-guided mask generation and points-mask fusion. The former submodule is responsible for generating centerline instance mask, guided by the centerline points, while the latter submodule further refines the centerline instance points (represented as green dots in Fig. 1), based on the predictions from the mask. It is important to note that virtual centerlines lack significant visual information, which makes the predicted mask for them less accurate for refining the centerline points. Hence, for virtual centerlines, we solely generate a mask for supervision, without refining their points based on the mask. However, for real centerlines, we obtain refined points denoted as $\mathbf{L}_{R}^{\prime}=\{\mathbf{l}_{1}^{\prime},\dots,\mathbf{l}_{N_{R}}^{\prime}\}$.

We categorize the ground truth centerlines into two categories: real and virtual. To construct losses, we perform bipartite matching separately for the virtual centerlines predicted in Section 3.2 and the real centerlines obtained from Section 3.3 with each category of ground truth. The final form of our loss function is presented as follows:

where we only descirbe the predictions in the loss function without explicitly mentioning the ground truth for brevity. The first two losses, denoted as $\mathcal{L}_{top}(\mathbf{A}_{ll})$ and $\mathcal{L}_{\text{cls}}(\mathbf{S})$, are implemented using focal loss. These losses are utilized to supervise the topological associations and centerline probabilities, respectively. The detection term, denoted as $\mathcal{L}_{\text{det}}(\mathbf{L}_{V},\mathbf{L}_{R})$, is computed using the L1 loss, which is employed to supervise the geometric shape of centerlines. To supervise the generation of the mask $\mathbf{M}$ for both real and virtual centerlines, we use the loss function $\mathcal{L}_{\text{mask}}(\mathbf{M})$, which combines the binary cross-entropy loss and the dice loss[23]. Furthermore, we employ the L1 loss, the binary cross-entropy loss, and the focal loss to supervise the column-based sampling mask points $\mathbf{C}$, the existence probabilities $\mathbf{P}$, and the points direction $D$ in the function $\mathcal{L}_{mp}(\cdot)$, respectively. In addition, we also add supervision for the row-based sampling mask points, following the same approach as the column-based ones. However, for brevity, this supervision is not explicitly stated in the equation.

We conduct a comparative evaluation of RoadPainter against current state-of-the-art (SOTA) methods on the OpenLane-V2 dataset[32]. The results are reported in Table 1 and Table 2, where we ensure a fair comparison by employing the same backbone for all methods. Our proposed method, RoadPainter, achieves an OLS score of 38.9, surpassing the performance of other SOTA methods on subset_A dataset. In comparison to alternative approaches, RoadPainter demonstrates superior results in terms of $\text{DET}_{l}$ (+2.2) and $\text{TOP}_{ll}$ (+0.7) metrics, highlighting the effectiveness of our model design. We also conducted experiments on the OpenLane-V2 subset_B dataset. Our approach outperforms others in centerline detection $\text{DET}_{l}$ and topology $\text{TOP}_{ll}$. We employ a 3-layer BEVFormer architecture, similar to TopoNet, while TopoMLP utilizes a 6-layer PETR[20] architecture. The PETR architecture is particularly suitable for traffic sign detection and achieves superior results on metrics such as $\text{DET}_{t}$ and $\text{TOP}_{lt}$. Moving forward, our main focus will be on enhancing traffic sign detection and topology association. Furthermore, by incorporating the SD map, our enhanced model, RoadPainter^∗, exhibits notable improvements in $\text{DET}_{l}$ (+6.2), $\text{TOP}_{ll}$ (+4.8) and OLS (+3.7) metrics. RoadPainter^∗ notably outperforms SMERF^∗ in both centerline detection and topology reasoning performance by a large margin. The OpenLane-V2 subset_B dataset does not include an SD map, so there are no available results for $\text{RoadPainter}^{*}$. The FPS of our model (FP32) on the RTX3090 is 6.5.

We conduct ablation studies on the point-guided mask generation (PGM), point-mask fusion (PMF), and SD map interaction (SD) modules using the subset_A dataset. Our analysis focuses on quantifying the impact of each module, with the detailed improvements outlined in Table 3. In addition, we extend our assessment to include the instance segmentation performance of centerline. This evaluation is based on the AP (average precision) metric, represented as $\text{AP}_{l}$.

Baseline. We construct the baseline model by retaining the vanilla centerline decoder without hybrid attention and real-virtual separation. In this setup, we utilize self-attention and deformable cross-attention mechanisms while omitting masked cross-attention. For supervision, we only incorporate the first three losses as outlined in Equation 8. Our baseline model achieves an OLS score of 37.2. Notably, the $\text{TOP}_{ll}$ metric of the baseline model (7.7) surpasses that of TopoNet (4.1), underscoring the efficacy of our topological association design.

Point-Guided Mask Generation. Building upon the baseline model, we incorporate the PGM module to infer centerline masks guided by the regressed points. When comparing the PGM model to the baseline, we observe a notable improvement of $1.2$ in $\text{DET}_{l}$ and $0.2$ in $\text{TOP}_{ll}$. This significant enhancement can be attributed to our innovative design of point-guided mask generation.

Point-Mask Fusion. We introduce a point-mask fusion module to enhance the accuracy of real centerlines by refining the points based on the generated masks. We observe improvements in the $\text{DET}_{l}$, OLS, and $\text{AP}_{l}$ metrics, with increases of $2.6$, $1.3$, and $0.6$, respectively. These results highlight the efficacy of the PMF module in refining real centerlines, thereby benefiting centerline detection, topology prediction, and mask generation.

SD Map Interaction. We employ the SD map interaction module to enhance the BEV features, thereby improving the accuracy of centerline detection and topological reasoning. By incorporating information from the SD map, our final model exhibits significant performance gains compared to previous models that do not utilize the SD map. Specifically, there is a substantial improvement in $\text{DET}_{l}$ (+6.2) and $\text{TOP}_{ll}$ (+4.8).

To better understand the attention mechanism in our method, we exclude masked attention in the transformer decoder (w/o hybrid attention) and omit real-virtual separation-based self-attention strategy (w/o real-virtual self-attn). As depicted in Table 4, the elimination of hybrid attention results in a slight performance decrease across all metrics. Furthermore, the absence of the real-virtual self-attention strategy notably affects the $\text{TOP}_{ll}$ metric (-0.4), indicating the contribution of the real-virtual separation-based self-attention approach.

To further verify the segmentation mask design in our method, we conduct three ablatin experiments: only detection, segmentation as auxiliary supervision, and segmentation as centerline sampling, as shown in Table 4. The findings indicate that segmentation, when used independently for auxiliary supervision and mask point sampling, can enhance the performance of centerline detection. Our method amalgamates these two benefits, thereby achieving superior results.

We present a comparison of centerline visualization results among TopoNet, RoadPainter, RoadPainter^∗ (with SD map), and ground truth (GT) in Fig. 3. The findings illustrate that RoadPainter exhibits superior lane detection and topological reasoning capabilities, particularly in complex scenarios such as lane transitions and intersections. RoadPainter^∗ demonstrates the ability to accurately predict centerlines and topological relationships beyond the visual range, facilitated by the SD map interaction module. In Fig. 4, we further present visualization results of centerlines from Section 3.2, along with instance masks and refined centerlines from Section 3.3. These results clearly demonstrate that, under the guidance of the instance mask, the refined centerlines exhibit superior performance in terms of accuracy, particularly at the junctions where details are crucial. In Fig. 5, our method exhibits superior performance compared to TopoNet in accurately extracting high-curvature lanes, which can be attributed to the utilization of segmentation masks. These findings indicate that our method brings notable improvements in lane detection, particularly in the accurate extraction of high-curvature lanes.