Holistic Autonomous Driving Understanding by Bird’s-Eye-View Injected
Multi-Modal Large Models

Our NuInstruct is built on one of the most popular datasets, i.e., Nuscenes [2]. There are six view records for samples of Nuscenes, i.e., Front, Front Left, Front Right, Back Right, Back Left, and Back. These views have some areas of overlap with one another. In NuScenes, the collected data is annotated with a frequency of $2$Hz, and each annotated frame is referred to as a keyframe with annotations.

In our research, we propose an SQL-based approach for the automated generation of four types of instruction-follow data, namely: Perception, Prediction, Risk, and Planning with Reasoning. This methodology aligns with the sequential decision-making stages of human drivers, categorized as follows: 1. Perception: The initial stage of recognizing surrounding entities. 2. Prediction: Forecasting the future actions of these entities. 3. Risk: Identifying imminent dangers, such as vehicles executing overtaking manoeuvres. 4. Planning with Reasoning: Developing a safe travel plan grounded in logical analysis.

The detailed process is shown in Fig. 2. Specifically, 1). The filter & construction step leverages the (a) original annotations to generate the scene information database (see Fig. 2 (b)). 2). The sampling step first samples three keyframes from the original dataset. Then, as shown in Fig. 2 (c)), we construct a series of pre-defined task SQLs. Each task SQL consists of several subtasks, each of them consisting of a subtask function and an instruction prompt. 3). The retrieval step uses the instruction prompt and the task SQL to retrieve the corresponding response from the scene database. 4). The saving step saves all instruction-response pairs (see Fig. 2 (d)). 5). The verifying step employs either human analysis or LLM-based methods (e.g., GPT-4 [35]) to eliminate erroneous instruction-response pairs, thereby guaranteeing the quality of NuInstruct. Our task SQL design is logically sequenced, and based on the inherent relational flow of autonomous driving tasks, i.e., ‘Perception $\rightarrow$ Prediction, (Perception, Prediction) $\rightarrow$ Risk, (Risk, Prediction) $\rightarrow$ Planing with Reasoning’, where $a\rightarrow b$ indicates the $b$ SQL is derived from the $a$ SQL (blue dashed arrows in Fig. 2 (c)).

We show a more detailed example for Step 2 and Step 3 in Fig. 3. Specifically, three keyframes (i.e., from frame $i$ to frame $i+2$) with annotations are randomly sampled from the original dataset, and we only select one instance, i.e., the pedestrian (box) for clarity (Fig. 3 (a)). In this case, we choose distance, status, crossing, and planning subtask SQLs from the perception, prediction, risk, and planning with reasoning task SQLs, respectively (shown in Fig. 3 (b)). The status subtask is based on the distance task, since the next status (e.g., speed, direction) for the instance is computed based on the distances of previous, current, and future frames. Each subtask SQL consists of two parts, i.e., the subtask function and the instruction prompt. For example, the distance subtask SQL has Find_Distance($<$instance$>$) function and the instruction prompt is ”What is the distance between $<$instance$>$ and the ego car? Please use the format as (x,y) where..”, where $<$instance$>$ is the input. Finally, as shown in Fig. 3 (c), we use the instance information or frame information as the input for different subtask functions to retrieve the response from the scene database. Compared with other data generation methods, e.g., ChatGPT-based [48] or human-based [10], this structured design ensures the generation of instruction-response pairs is both reliable and scalable.

We only describe the overview of the data generation in this section. Please refer to the supplementary material for more details about the scene information database, the task SQLs, and the retrieval process.

To construct our NuInstruct, we sampled a total of $11,850$ keyframes from $850$ videos within the NuScenes dataset [2]. Subsequent filter yields $55,204$ unique instances, which collectively appear $295,828$ times across all keyframes. This culminates in an average of approximately $24.96$ instances per keyframe. By employing our SQL-based method (Section 3.1), we generated a total of $91,355$ instruction-response pairs, encompassing four primary tasks—namely, Perception, Prediction, Risk, and Planning with Reasoning. These tasks are further delineated into $17$ sub-tasks. The quantities of task categories are statistically presented in Fig. 4 (a).

Compared with other single-view benchmarks, our dataset covers multi-view information. Hence, we also conduct a statistical analysis of the relations of different views and constructed instruction-response pairs. Fig. 4 (b) shows the distribution of the numbers of the responses based on the views, i.e., for a given view, we record how many responses are derived from information from the view. Through such statistics, we find that to answer the instructions, the system needed to look at multiple views, instead of just a single one. In Fig. 4 (c), for each task, the proportions of responses obtained based on different views are calculated. We find two observations: (i) in the case of perception and prediction tasks, the distribution across views is relatively even, showing that our data generation method produces balanced multi-view information; (ii) When it comes to reasoning and risk tasks, the responses predominantly draw on information from the front, left-front, and right-front views. This is reasonable since drivers normally base their ahead or sides to decide the next actions, seldom needing to look behind.

Different from current MLLMs, the visual inputs for our model are the multi-view videos $\{\mathbf{V}^{i}\}_{i=1}^{N_{\text{view}}}$, where $N_{\text{view}}$ is the total number of camera views, $\mathbf{V}^{i}=\{\mathbf{v}^{i}_{t}\}_{t=1}^{N_{\text{frame}}}$, $\mathbf{v}^{i}_{t}$ is the $t$-th frame in $\mathbf{V}^{i}$ and $N_{\text{frame}}$ is the total number of frames. Instead of the predefined several tasks, we give a specific language instruction, we use a unified model to obtain its corresponding language response, as shown in Fig. 1. For clarity in the following, we use $\mathbf{L}_{\text{inst}}\mathbb{R}^{N_{\text{inst}}\times D_{\text{inst}}}$ and $\mathbf{L}_{\text{resp}}\in\mathbb{R}^{N_{\text{resp}}\times D_{\text{resp}}}$ to denote the language instruction tokens and the response tokens respectively, which are generated by the language tokenizer [44]. $N_{\text{inst}}$/$N_{\text{resp}}$ and $D_{\text{inst}}$/$D_{\text{resp}}$ are numbers of tokens and dimensions for the instruction/response.

Existing MLLMs [50, 23, 7, 25] generally consist of three parts: a vision encoder $f_{\text{vision}}(\cdot)$ to receive the visual input; a connection module (e.g., Q-Former [24]) $f_{\text{connect}}(\cdot)$ to transfer the visual representations to the visual tokens aligned with the language; a large language model (LLM) $f_{\text{LLM}}(\cdot)$ to receive visual and language instruction tokens to generate the response. Since they can only receive single-view input, we propose a baseline model named Multi-view MLLM (MV-MLLM) to enable the current MLLMs to process the multi-view videos, as shown in Fig. 5 (a). Specifically, for a video from a specific view, i.e., $\mathbf{V}^{i}$, we feed it into the vision encoder followed by the connect module to obtain the visual tokens, which can be formulated as:

where $N_{\text{vis}}$ and $D_{\textbf{v}}$ are the numbers of the visual tokens and the dimensions respectively. Then, we introduce a multi-view Q-Former (similar to BLIP-2 [23]) to capture the MV visual semantics $\mathbf{F}_{\text{mv}}$ from $\{\mathbf{F}^{i}_{\text{vis}}\}_{i=1}^{N_{\text{view}}}$. We concatenate $\{\mathbf{F}^{i}_{\text{vis}}\}_{i=1}^{N_{\text{view}}}$ along the view dimension to obtain $\overline{\mathbf{F}}_{\text{mv}}\in\mathbb{R}^{(N_{\text{view}}*N_{\text{vis}})\times D_{\text{vis}}}$. The input to the multi-view Q-Former contains a set of $K_{\text{mv}}$ learnable multi-view queries $\mathbf{Q}_{\text{mv}}\in\mathbb{R}^{K_{\text{mv}}\times D_{\text{vis}}}$, which interact with $\overline{\mathbf{F}}_{\text{mv}}$ through the cross attention, formulated as follows:

The output $\mathbf{F}_{\text{mv}}$ then goes through a linear projection (omitted in Fig. 5), and is fed to the LLM. Note that in our MV-MLLM, only the MV Q-Former is trainable and other parameters are frozen to fully retain the knowledge of the pre-trained models.

The BEV approach has become pivotal in autonomous driving for its precise depiction of object positioning, essential for tasks like perception and planning [32, 26]. Integrating BEV into our MV-MLLM offers enhanced visual-spatial analysis for autonomous driving. While BEV can be constructed from the multi-view features $\{\mathbf{F}^{i}_{\text{vis}}\}_{i=1}^{N_{\text{view}}}$ by physical transformations such as LSS [38], similar to NuScenes-QA [41], the plain of ViTs in current MLLMs limits their perception capabilities [36]. Replacing the VIT with BEV-specific backbones, e.g., ResNet [14] or Swin Transformer [28], diminishes visual-language alignment [11]. Furthermore, the limited input resolutions generate small feature maps, which are hard to scale up to the high-resolution BEV representation.

To address the above problems, we propose the BEV-injected MLLM (BEV-InMLLM), which uses a BEV injection module (BEV-In) to obtain the BEV information aligned with LLMs in a data-efficient and resource-light way. As shown in Fig. 5 (b), we obtain the high-quality BEV features $\mathbf{F}_{\text{bev}}\in\mathbb{R}^{W\times H\times D_{\text{bev}}}$ from the pre-trained BEV extractor [38, 17], where $W$, $H$ and $D_{\text{bev}}$ denote the width, height and dimensions. The following are two key components of BEV-In, i.e., an instruction-aware BEV Q-Former and an injection module.

We select three advanced MLLMs, i.e., BLIP-2 [24], MiniGPT-4 [50] and Video-LLama [49] as our base models. For each MLLM, we apply our proposed modules to obtain MV-MLLM and BEV-InMLLM. All models are finetuned in the same setting. We report our results on NuInstruct test set in Table 3. To conserve space, we aggregate the reporting of two subtasks, ‘motion ego’ and ‘motion others’. A similar approach is adopted for ‘status ego’ and ‘status others’.

From Table 3, we observe that equipped with our proposed modules, there is a significant increase in the evaluation metrics on all tasks, demonstrating its effectiveness. More specifically, the integration of temporal and multi-view information (MV-MLLM) substantially improves risk and planning tasks by $~{}5\%$ and $~{}6\%$, respectively. Furthermore, injecting BEV into MV-MLLM, i.e., BEV-InMLLM, benefits tasks sensitive to distance and position, e.g., perception and prediction.

In this section, we conduct experiments to evaluate the effect of different proposed modules and different input information. Here, we use MiniGPT-4 [50] as our baseline model. To better analyze the impact of each module on autonomous driving tasks, we reclassify the sub-tasks into four main tasks based on their dependency on different types of information. These are categorized as temporal-related (temporal), multi-view-related (multi-view), spatial-related (Spatial), and holistic-related tasks, as shown in Table 4. ‘Temporal’ indicates subtasks related to temporal cues, e.g., the vehicle’s status is determined based on its positions at various times, and so do others. Note that some subtasks may be classified into different tasks, e.g., the status task is in both temporal and spatial tasks. We will report the results of different models under the reclassified tasks in the following.

Based on the information of the ego car and important instances, we construct a scene database as shown in Fig. 6 (a). The scene information database includes four tables:

• •

  Scene Information: consists of the scene dictionary list. The key for each dictionary is the scene ID, which is the unique identifier of the scene in NuScenes [2]. The value is a frame ID list, consisting of IDs of frames in the current scene. The details for each frame are referred to the Frame Information table.

• •

  Frame Information: consists of the frame dictionary list. The key for each dictionary is the unique ID for a specific frame. The value contains the details for the frame, including the ego-car-information ID and instance-information ID list. The details for the information of the ego car and instances in the frame are referred to the Ego Information and Instance Information tables respectively.

• •

  Ego Information: consists of the ego-car information dictionary list. The key for each dictionary is the unique ID for the information of the ego-car in one specific frame. The values include the information:  e.g., ego car pose, ego car rotation, velocity, road information, camera information, and so on.

• •

  Instance Information: consists of the instance information dictionary list. The key for each dictionary is the unique ID for the information of one instance in one specific frame. The values include the information:  e.g., instance ID (the unique identifier for one instance, e.g., a car, across different frames in one scene), instance global and local translations and rotations, velocity, road information, etc.

The relations between different tables in the database are shown in Fig. 6 (b) (red dashed arrows), i.e., Scene Information and Frame Information, Frame Information and Ego Information, Frame Information and Instance Information are all one-to-many mappings. The detailed fields and their corresponding types for each table are shown in Table 7(d).

As shown in Fig. 6 (b), we define four types of SQL sets (i.e., perception, prediction, risk, and planning with reasoning), which are used for generating different types of instruction-response pairs. Each type of task SQL set consists of several subtask SQLs. For example, the perception SQL set contains six subtask SQLs, e.g., Closest focuses on finding the objects closest to the ego car, and the other subtasks are similar. Note that some high-level SQLs may be inherited from low-level ones following the relational flow of autonomous driving tasks [16], as shown in blue dashed arrows in Fig. 6 (b). For instance, the prediction SQLs are based on the perception ones, the risk SQLs are based on both prediction and perception SQLs, and the planning with reasoning SQL is inherited from prediction and risk ones.

Each subtask SQL consists of a subtask function and an instruction prompt. We illustrate the detailed algorithms for $17$ subtask SQLs in Algorithm 1-17. In these algorithms, ‘query($T$, *args)’ indicates querying the information from the table $T$ with the arguments *args. After obtaining the results from the task SQLs, we transfer them into the language descriptions by template or GPT-4 [35]. Regard the planning with reasoning task as the example, after obtaining the queried results $\{R,s,m\}$, where $R$, $s$ and $m$ are the risk instance dictionary, the future speed and the future motion of the ego car. Then, the response is formulated as ‘There are $R$ the ego car. Hence the ego car should be $s$ and move to $m$.’

In this section, we show the computation details for different evaluation metrics, i.e., MAE, accuracy, MAP, and BLEU.