OccProphet: Pushing Efficiency Frontier of Camera-Only 4D Occupancy Forecasting with Observer-Forecaster-Refiner Framework

Occupancy prediction aims at modeling the current 3D occupancy layout in space, by observing historical and current environments. Occupancy prediction is adept at providing 3D dense descriptions for complex traffic scenarios, thereby garnering increasing attention from academia and industry. SSCNet (Song et al., 2017) was the first semantic occupancy prediction work, which simultaneously predicted occupied voxels and their semantics for an indoor scene using a depth image. MonoScene (Cao & De Charette, 2022) extends SSCNet to outdoor scenarios by using an RGB image and incorporating stronger supervisions. Training and evaluating occupancy prediction networks require benchmarks with ground truth occupancy labels, which are challenging due to the complexity of densely annotating 3D outdoor driving scenes. Wang et al. (Wang et al., 2023b) propose OpenOccupancy, the first large-scale benchmark for semantic occupancy prediction, which covers multiple sensing modalities and provides high-resolution dense occupancy annotations. Occ3d, developed by Tian et al. (Tian et al., 2023), is another widely used benchmark for occupancy prediction, whose high-quality labels are from the proposed technique of image-guided occupancy label refinement.

Considering the inherently dense nature of depicting 3D environments, ideal occupancy perception emphasizes balancing efficiency and accuracy. Although some studies explore sparse queries (Li et al., 2023b) or tri-perspective view (TPV) representation (Huang et al., 2023a) to elevate the efficiency of occupancy prediction, they inevitably sacrifice fine-grained details of 3D space. In contrast, many other methods (Zhang et al., 2023; Wei et al., 2023b; Ma et al., 2024b) utilize 3D feature volumes to preserve 3D details of the scene, leading to higher occupancy prediction accuracy. Recently, COTR (Ma et al., 2024b) proposes a compact occupancy representation that preserves geometric details while reducing computational costs. Despite the significant advancements in occupancy prediction, including comprehensive benchmarks and powerful algorithms, all existing methods focus exclusively on current occupancy and overlook future occupancy, which reflects potential variations in the 3D environment.

Occupancy forecasting targets to predict future occupancy, starting from the current timestamp. Previous dominant works mainly adopt the BEV perspective for occupancy forecasting, reasoning about 2D occupancy changes on a BEV plane. For example, FIERY (Hu et al., 2021) extracts BEV features from multi-view image inputs and utilizes a temporal model with 3D convolution to capture spatio-temporal states, which are then used to recursively forecast future instances states. BEVerse (Zhang et al., 2022) introduces a unified BEV representation framework that jointly achieves object perception and occupancy forecasting using multi-task supervision. To better align spatio-temporal information, TBP-Former (Fang et al., 2023) designs a pose-synchronized BEV encoder to synchronize multi-frame BEV features during occupancy forecasting. While these BEV-based methods deliver impressive performance for forecasting pre-defined semantic categories (e.g., vehicles), they struggle to (1) forecast the motion of out-of-distribution objects and (2) capture height information in the environment. In contrast, our method forecasts class-agnostic occupancy from a 3D perspective, rather than BEV, thus enabling autonomous vehicles to monitor, comprehend, and reason about 3D dynamics in the physical world.

To address the limitations of BEV occupancy forecasting, researchers have recently shifted their focus toward forecasting 3D occupancy without considering semantics. Specifically, Khurana et al. (Khurana et al., 2023) treat LiDAR point cloud forecasting as a proxy task for the occupancy forecasting task, where point cloud rendering is used to bridge the two tasks. UnO (Agro et al., 2024) takes LiDAR point clouds as input and performs occupancy forecasting using the proposed unsupervised learning paradigm, in which the forecasted occupancy should align with pseudo occupancy labels generated from future LiDAR data. However, these methods rely on point clouds from expensive LiDAR kits, leading to increased costs when implemented in autonomous vehicles.

Compared to LiDAR-based approaches, camera-only occupancy forecasting offers a promising alternative with significantly lower costs. Cam4DOcc (Ma et al., 2024a) introduces a comprehensive benchmark and dataset to evaluate camera-only occupancy forecasting algorithms on both movable and static objects beyond pre-defined categories. Additionally, it proposes a strong camera-only baseline for occupancy forecasting. However, this approach is still far from real-world application due to its high computational demands. Drive-OccWorld (Yang et al., 2024) introduces extra action condition inputs and planning supervision to enhance performance. In this paper, we propose a novel end-to-end framework with faster speed, and higher accuracy, enabling efficient and effective occupancy forecasting in a pure camera-only setting.

The overview of the proposed OccProphet is illustrated in Figure 3. Given multi-frame surround-view RGB images as input, 2D features are extracted using a shared image encoder. These 2D features are subsequently lifted into 3D space and aggregated into multi-frame 3D voxel features through depth estimation and voxel pooling. We design the following unshared pipeline for each of the occupancy and occupancy flow branches. Specifically, the pipeline consists of four components: the Observer, Forecaster, Refiner, and Predictor. The Observer module efficiently and effectively aggregates spatio-temporal information within multi-frame observations (i.e., multi-frame 3D voxel features). The Observer’s output then undergoes a Forecaster, which adaptively predicts future states, ensuring flexibility across diverse traffic conditions. The Refiner module further enhances the quality of these predictions by enabling cross-frame interactions. Finally, the Predictor module decodes the refined future states into either occupancy or occupancy flow.

The Observer takes the 4D motion-aware feature $F_{motion}$ as input, and generates a spacetime-aware representation. Let $\mathcal{I}_{t}=\{I_{t}^{v}\}_{v=1}^{N_{cam}}$ denote the multi-camera RGB images at timestamp $t\in\{1,\dots,T\}$, where $N_{cam}$ refers to the total number of surround-view cameras, and $T$ is the total number of input frames. A shared image encoder (i.e., ResNet (He et al., 2016)) is applied to $\{\mathcal{I}_{t}|t=1,\dots,T\}$ to extract 2D features. For each frame, these 2D features are projected to 3D space and then aggregated into voxelized 3D features (Philion & Fidler, 2020). The 3D features from multiple frames are aligned into the current-frame coordinate system using 6 degrees of freedom (6-DoF) ego-vehicle poses. We then concatenate the aligned 3D features into a 4D feature $F\in\mathbb{R}^{T\times C\times X\times Y\times Z}$, where $C$ is the number of channels, and $(X,Y,Z)$ represents the size of voxelized 3D feature volume. Subsequently, the motion-aware 4D feature $F_{motion}\in\mathbb{R}^{T\times(C+6)\times X\times Y\times Z}$ is generated by concatenating 6-DoF ego-vehicle poses.

To generate the spacetime-aware representation, directly processing the 4D motion-aware feature using convolutional operations is intuitive. However, the direct processing imposes a substantial computational burden, and ignores the fact that a large portion of the 3D space is unoccupied, which leads to the inherent sparsity of the motion-aware feature. To address this issue, we efficiently and effectively generate the spacetime-aware feature from $F_{motion}$ using the Observer module. The Observer comprises an Efficient 4D Aggregation module and a Tripling-Attention Fusion module.

Given a spatio-temporal representation $O_{\text{obs}}\in\mathbb{R}^{T\times C\times X\times Y\times Z}$ output from the Observer, the Forecaster (shown in Figure 6) is supposed to generate future states. We first reshape $O_{\text{obs}}$ by collapsing the time axis into the channel axis, resulting in the reshaped features $O_{\text{obs}}^{\prime}\in\mathbb{R}^{TC\times X\times Y\times Z}$. A straightforward approach to forecasting is using a single linear layer to predict features for future frames. This approach encourages the network to learn static weights to fit different traffic scenarios. However, traffic situations vary significantly across different traffic scenarios, presenting distinct challenges in predicting future changes using a single linear layer. For instance, forecasting environmental changes is more difficult in a crowded intersection with many moving objects than on an empty highway with few vehicles. The spatio-temporal complexity in the former is far greater than in the latter.

Considering these challenges, we propose forecasting occupancy with flexibility to adapt to various traffic scenarios featuring diverse spatio-temporal complexities. To achieve this, we design a novel Forecaster module that predicts future states based on the overall environmental condition. The Forecaster comprises a Condition Generator and a Conditional Forecaster. We first use a Condition Generator comprising a 3D GAP and a shared linear layer across different frames to extract the condition from the observation $O_{\text{obs}}$:

where $F_{\text{cond}}^{\prime}\in\mathbb{R}^{T\times C\times 1\times 1\times 1}$ denotes the overall environmental condition, which is then reshaped into $F_{\text{cond}}\in\mathbb{R}^{TC}$ by collapsing the time axis into the channel axis. $F_{\text{cond}}$ is subsequently passed to a Conditional Forecaster to predict future states. Specifically, a linear layer is applied to $F_{\text{cond}}$ to produce adaptive weights for specific traffic scenarios. Another linear layer is then used to predict future states conditioned by these adaptive weights.

where $W_{cond}\in\mathbb{R}^{TC\times T^{\prime}C}$ denotes the adaptive weights, $F_{\text{future}}^{\prime}\in\mathbb{R}^{T^{\prime}C\times X\times Y\times Z}$ is the future states of $T^{\prime}$ future frames. $F_{\text{future}}^{\prime}$ is reshaped into $F_{\text{future}}\in\mathbb{R}^{T^{\prime}\times C\times X\times Y\times Z}$ to recover the time axis. $F_{\text{future}}$, as the preliminary feature for future environments, is then refined by the Refiner module.

Since the Forecaster module predicts $F_{\text{future}}$ using linear projection, it inevitably lacks cross-frame interactions. The Refiner is designed to enhance the forecasted results via further interactions between future frames, as well as incorporating historical frames as supplementary information. The E4A module, described in Section Section 3.2.1, is a spatio-temporal interaction module. Furthermore, taking a review of the E4A, for any input feature $Q\in\mathbb{R}^{T\times C\times X\times Y\times Z}$, the function of the E4A module can be formulated as:

where $Q^{\prime}\in\mathbb{R}^{T\times C\times X\times Y\times Z}$ is the output feature of E4A, and $\mathcal{F}$ denotes the transformation function. Considering residual networks can aid in refinement and network optimization (He et al., 2016), it is reasonable to regard E4A as a refinement transformation for features, which also reduces the learning complexity of earlier modules. Building on this insight, we further introduce a Refiner reusing the E4A architecture to refine the forecasted future states. The Refiner is applied to the forecasted feature $F_{\text{future}}$ from the Forecaster and the feature output $O_{\text{obs}}$ from the Observer, producing an enhanced feature $V=E4A([O_{\text{obs}},F_{\text{future}}])_{T+1:T+T^{\prime},:,:,:,:}\in\mathbb{R}^{T^{\prime}\times C\times X\times Y\times Z}$ for subsequent forecasting of occupancy and flow.

For fair comparisons, the Predictor for occupancy and occupancy flow prediction, and the overall training loss function follow those in Cam4DOcc (Ma et al., 2024a).

In this section, we compare the model complexity between Cam4DOcc and OccProphet. As shown in Table 6, the number of parameters, memory usage, and FLOPs of OccProphet are decreased by 58-78% compared to Cam4DOcc, while OccProphet achieves a 4% relative increase in $\tilde{\text{IoU}}_{f}$, and has 2.6$\times$ the FPS speed of Cam4DOcc, justifying the efficiency and effectiveness of OccProphet.