SODFormer: Streaming Object Detection with Transformer Using Events and Frames

The publicly available object detection datasets utilizing event cameras have a limited amount [29, 30, 31, 32]. Gen1 Detection dataset [33] and 1Mpx Detection dataset [34] provide large-scale annotations for object detection using event cameras. However, it may be difficult to obtain fine textures and achieve high-precision object detection by only using DVS events. Although event-based simulators (e.g., ESIM [35], V2E [36], and RS [37]) can directly convert video object datasets to the neuromorphic domain, those transformed strategies fail to reflect realistic high-speed motion or extreme light scenarios, which are exactly what event cameras are good at. Besides, PKU-DDD17-CAR dataset [14] and DAD dataset [16] provide 3155 and 6247 isolated hybrid sequences, respectively. Nevertheless, these two small-scale datasets are not continuous streams modeling temporal dependencies. Therefore, this work aims to establish a large-scale and temporally long-term multimodal object detection dataset involving challenging scenarios.

The existing object detectors using event cameras can be roughly divided into two categories. The first category [38, 39, 40, 41, 42, 43, 44, 45, 46] is the single-modality that only processes asynchronous events. These methods usually first convert asynchronous events into 2D image-like representations [47], and then input them into frame-based detectors (e.g., YOLOs [48]). Although only utilizing dynamic events can achieve satisfactory performance in some specific scenes, it becomes clear that high-precision detection requires static fine textures.

The second category [49, 14, 13, 6, 15, 16, 27, 50, 51] refers to the multi-modality that combines multiple visual streams. For example, some works [49, 14, 13] first detect objects on each isolated frame or event temporal bin, and then merge the detection results of two modalities by post-processing (e.g., NMS [52]). Besides, a grafting algorithm [6] is proposed to integrate events and thermal images. Some attention-based aggregation operations [15, 27] are designed to fuse each isolated frame and event temporal bin from the PKU-DDD17-CAR [14] dataset, while others [16, 28] average the features from each modality. However, these joint frameworks have not explored rich temporal cues from continuous visual streams. Moreover, these post-processing or aggregation fusion operations are hard to capture global context across two sensing modalities, while averaging strategy cannot eliminate unimodal degradation thoroughly. Thus, we design a streaming object detector that aims at leveraging rich temporal cues and making fully complementary use of two visual streams.

Transformer, an attention-based architecture, is first introduced by [22] for machine translation. Its core attention mechanism [53] scans through each element of a sequence and updates it by aggregating information from the whole sequence with different attention weights. The global computation makes Transformers perform better than RNNs on long sequences. Until now, Transformers have been migrated to computer vision and replaced RNNs in many problems. In video object detection, while RNNs [54, 55, 56] have achieved great success, they require meticulously hand-designed components (e.g., anchor generation). To simplify these pipelines, DETR [20], an end-to-end Transformer-based object detector, is proposed. It is worth mentioning that DETR [20] is the first Transformer-based object detector via a sequence predicting model. While DETR largely simplifies the classical CNN-based detection paradigm, it suffers from very slow convergence and relatively low performance at detecting small objects. More recently, a few approaches attempt to design optimized architectures to help detect small objects and speed up training. For instance, Deformable DETR [21] adopts the multi-scale deformable attention module that achieves better performance than DETR with 10$\times$ fewer training epochs. PnP-DETR [57] significantly reduces spatial redundancy and achieves more efficient computation via a two-step poll-and-pool sampling module. However, most works operate on each isolated image and do not exploit temporal cues from continuous visual streams. Inspired by the ability of Transformer to model long-term dependencies in video sequence tasks [24, 25, 26], we propose a temporal Transformer model to leverage rich temporal cues from continuous events and adjacent frames.

Some computer vision tasks (e.g., video reconstruction [12, 58, 59, 60], object detection [14, 13], object tracking [61, 62], depth estimation [11], and SLAM [63, 64, 65]) have sought to integrate two complementary modalities of events and frames. For example, JDF [14] adopts the Dempster-Shafer theory to fuse events and frames for object detection. A recurrent asynchronous multimodal network [11] is introduced to fuse events and frames for depth estimation. In fact, most fusion strategies (e.g., post-processing, concatenation, and averaging) can’t distinguish the degradation degree of different modalities and regions, resulting in the failure to eliminate unimodal degradation completely (see Section 2.2). More importantly, most fusion strategies are synchronized with event and frame streams, which causes the joint output frequency to be limited by the sampling rate of conventional frames. Obviously, the limited inference frequency can’t meet the needs of fast object detection in real-time high-speed scenarios, so we seek to utilize the high temporal resolution advantage of event cameras and make use of events and frames. In RAMNet [11], RNN is utilized to introduce an asynchronous fusion method that allows decoding the task variable at any time. Nevertheless, to the best of our knowledge, there are no prior attention-based works involving asynchronous fusion strategy for events and frames in the object detection task. Thus, we design a novel asynchronous attention-based fusion module that eliminates unimodal degradation in two modalities with high inference frequency.

The goal of this dataset is to offer a dedicated platform for the training and evaluation of streaming object detection algorithms. Thus, a novel multimodal vision sensor (i.e., DAVIS346, resolution 346$\times$260) is used to record multiple hybrid sequences in a driving car (see Fig. 2). The event camera simultaneously outputs events with high temporal resolution and conventional frames with 25 FPS. Mostly, we fix the DAVIS346 in a driving car (see Fig. 2(2(a))), and record the sequences while the car is driving on city roads. Nevertheless, it is difficult to capture high-speed motion blur owing to the relative speed between vehicles on city roads. For the convenience of acquiring high-speed objects, we additionally provide some sequences in which the camera is set at the flank of the road. The raw recordings consider velocity distribution, light condition, category diversity, object scale, etc. To provide manual bounding boxes in challenging scenarios (e.g., high-speed motion blur and low-light), grayscale images are reconstructed from asynchronous events using E2VID [66] at 25 FPS when RGB frames are of low quality. When the objects are not visible in all three modalities (i.e., RGB frames, event images, and reconstructed frames), they can still be labeled because our PKU-DAVIS-SOD dataset provides continuous visual streams, and we can obtain information about the unclear objects from nearby images. Furthermore, some unclear objects in a single image may be visible in a piece of video. After the temporal calibration, we first select three common and important object classes (i.e., car, pedestrian, and two-wheeler) in our daily life. Then, all bounding boxes are annotated from RGB frames or synchronized reconstructed images by a well-trained professional team.

PKU-DAVIS-SOD dataset provides 220 hybrid driving sequences and labels at a frequency of 25 Hz. As a result, this dataset contains 276k timestamps (i.e., labeled frames) and 1080.1k bounding boxes in total. Afterward, we split them into 671.3k for training, 194.7k for validation, and 214.1k for testing. Table I shows the number of bounding boxes in each set. Besides, we further analyze the attributes of the newly built dataset from the four following perspectives.

Category Diversity. Fig. 3(3(a)) displays the number distributions of three types of labeled objects (i.e., car, pedestrian, and two-wheeler) in our PKU-DAVIS-SOD dataset. We can find that the numbers of cars, pedestrians, and two-wheelers in each set (i.e., training, validation, and testing) are (580340, 162894, 193118), (34744, 5800, 5680), and (67599, 28400, 19144), respectively. For intuition, the ratios of the above numbers are approximately 3.5 : 1 : 1.2, 6 : 1 : 1, and 3.5 : 1.5 : 1, respectively.

Object Scale. Fig. 3(3(b)) shows the proportions or object scales in our PKU-DAVIS-SOD dataset. The object height is used to reflect its scale since the height and the scale are closely related in driving scenes [67]. We broadly classify all objects into three scales (i.e., large, medium, and small). The medium scale varies from 20 pixels to 80 pixels, the small refers to the object less than 20 pixels in height, and the large scale means the object height is greater than 80 pixels.

Velocity Distribution. As illustrated in Fig. 3(3(c)), we divide the motion speed level into normal-speed and high-speed. Event camera, offering high temporal resolution, can capture high-speed moving objects, such as a rushing car. On the contrary, RGB frames may result in motion blur due to the limited frame rate. In the process of dividing the dataset, we use video as the basic unit. If a sequence contains lots of motion blur scenarios, it is classified as high-speed, otherwise, it would be classified as normal-speed instead. Fig. 3(3(c)) shows that the proportion of high-speed scenarios is 13%, and the remaining involving normal-speed objects is 87%.

Light Intensity. Since the raw visual streams are recorded from daytime to night, it covers the diversity in illumination variance. The sequences belonging to the low-light scenario are generally collected in low-light conditions (night, rainy days, tunnels, etc.) and their RGB frames consist mainly of low-light scenes. Thus, we can easily judge the light condition by viewing RGB frames with our eyes. Fig. 3(3(d)) illustrates that the proportions of normal light and low-light are 92% and 8%, respectively.

For better visualization, we show representative samples (see Fig. 4) including RGB frames, event images, and DVS reconstructed images, which cover the category diversity, object scale, velocity, and light change.

To clarify the advancements of the newly built dataset, we compare it with some related object detection datasets using event cameras in Table II. Apparently, our PKU-DAVIS-SOD dataset is the first large-scale and open-source multimodal neuromorphic object detection dataset ¹¹1https://www.pkuml.org/research/pku-davis-sod-dataset.html. In contrast, two publicly large-scale datasets (i.e., Gen1 Detection [33] and 1 Mpx Detection [34]) only provide temporally long-term event streams, thus it is difficult for them to serve high-precision object detection, especially in static or extremely slow motion scenarios. What’s more, TUM-Pedestrian dataset [13] and Pedestrian Detection dataset [41] provide event streams for pedestrian detection, but they have not yet been publicly available. DAD dataset [16], followed by PKU-DDD17-CAR dataset [14], offers isolated frames and event temporal bins rather than temporally continuous visual streams.

All in all, such a novel bio-inspired multimodal camera and professional design with high labor intensity enable our PKU-DAVIS-SOD to be a competitive dataset with multiple characteristics: (i) High temporal sampling resolution with 12 Meps from event streams; (ii) High dynamic range property with 120 dB from event streams, (iii) Two temporally long-term visual streams with labels at 25 Hz, (iv) Real-world scenarios with abundant diversities in object category, object scale, moving velocity, and light change.

DAVIS is a multimodal vision sensor that simultaneously outputs two complementary modalities of asynchronous events and frames under the same pixel array. Due to the complementarity of events and frames, the output of the DAVIS camera contains more information, making the DAVIS camera naturally surpass unimodal sensors like conventional RGB cameras and event-based cameras. In particular, the core event camera [68], namely DVS, has independent pixels that react to changes in light intensity $R(\bm{u},t)$ with a stream of events. More specifically, an event $e_{n}$ is a four-attribute tuple $(x_{n},y_{n},t_{n},p_{n})$ using addressing event representation (AER) [69], triggered for the pixel $\bm{u}=(x_{n},y_{n})$ at the timestamp $t_{n}$ when the log-intensity changes over the pre-defined threshold $\theta_{th}$. This process can be depicted as:

where $\Delta t_{n}$ is the temporal sampling interval at a pixel, the polarity $p_{n}\in\left\{-1,1\right\}$ refers to whether the brightness is decreasing or increasing.

Intuitively, asynchronous events appear as sparse and discrete points [70] in the spatiotemporal domain, which can be described as follows:

where $N_{e}$ is the event number during the spatiotemporal window. $\delta\left(\cdot\right)$ refers to the Dirac delta function, with $\int$$\delta\left(t\right)dt=1$ and $\delta\left(t\right)=0,\forall$$t\neq 0$.

Let $S(x,y,t)$ and $I$=$\left\{I_{1},...,I_{N}\right\}$ are two complementary modalities of asynchronous events and frame sequence from a DAVIS camera. In general, to make the asynchronous events compatible with deep learning methods, a continuous event stream needs to be divided into $K$ event temporal bins $S=\left\{S_{1},...,S_{K}\right\}$. For the current timestamp $t_{i}$, the spatiotemporal location information of objects (i.e., bounding boxes) can be calculated using adjacent frames $\left\{I_{i-n},...,I_{i}\right\},n\in[0,N]$ and multiple temporal bins $\left\{S_{i-k},...,S_{i}\right\},k\in[0,K]$, it can be formulated by:

where $B_{i}=\left\{\left(x_{i,j},y_{i,j},w_{i,j},h_{i,j},c_{i,j},p_{i,j},t_{i}\right)\right\}_{j\in[1,J]}$ is a list of $J$ bounding boxes in the timestamp $t_{i}$. More specifically, $(w_{i,j},h_{i,j})$ are the width and the height of the $j$ bounding box, $(x_{i,j},y_{i,j})$ are the corresponding upper-left coordinates, $c_{i,j}$ and $p_{i,j}$ are the predicted class and the confidence score of the $j$ bounding box, respectively. The function $\mathcal{D}$ is the proposed streaming object detector, which can leverage rich temporal cues from $n+1$ adjacent frames and $k+1$ temporal bins. The parameters $n$ and $k$ determine the length of mining temporal information and affect the fusion strategy between two heterogeneous visual streams.

Given the ground-truth $\bar{B}=\left\{\bar{B}_{1},...,\bar{B}_{N}\right\}$, we aim at making the output $B=\left\{B_{1},...,B_{N}\right\}$ of the optimized detector $\hat{\mathcal{D}}$ to fit $\bar{B}$ as much as possible, and it can formulate the following minimization problem as:

where $\mathcal{L_{\mathcal{D}}}$ is the designed loss function of the proposed streaming object detector.

Note that, this novel streaming object detector using events and frames works differently from the feed-forward object detection framework (e.g., YOLOs [48] and DETR [20]). It has two unique properties, which exactly account for the nomenclature of ”streaming”: (i) Objects can be accurately detected via leveraging rich temporal cues; (ii) Event streams offering high temporal resolution can meet the need of detecting objects at any time among continuous visual steams and overcome the limited inference frequency from conventional frames. Consequently, following similar works using point clouds [71, 72] and video streams [73], we call our proposed detector as streaming object detector.

This work aims at designing a novel streaming object detector with Transformer, termed SODFormer, which continuously detects objects in an asynchronous way by integrating events and frames. As illustrated in Fig. 5, our framework consists of four modules: event representation, spatial Transformer, temporal Transformer, and asynchronous attention-based fusion module. More precisely, to make asynchronous events compatible with deep learning methods, the continuous event stream is first divided into event temporal bins $S=\left\{S_{1},...,S_{K}\right\}$, and each bin $S_{i}$ can be converted into a 2D image-like representation $E_{i}$ (i.e., event embeddings [47]). Then, the spatial Transformer adopts the main structures of the Deformable DETR [21] to extract feature representations from each sensing modality, which involves a feature extraction backbone (e.g., ResNet50 [74]) and the spatial Transformer encoder (STE). Besides, the proposed temporal Transformer contains the temporal Deformable Transformer encoder (TDTE) and temporal Deformable Transformer decoder (TDTD). The TDTE assigns the outputs of STE in the temporal dimension, which can improve the accuracy of object detection by leveraging rich temporal information. Meanwhile, the proposed asynchronous attention-based fusion module exploits the attention mechanism to generate a fused feature, which can eliminate unimodal degradation in two modalities with a high inference frequency. Finally, The TDTD integrates object queries and the fused feature to predict the bounding boxes $B_{i}$.

When using deep learning methods to process asynchronous events, spatiotemporal point sets are usually converted into successive measurements. In general, the kernel function [47] is usually used to map the event temporal bin $S_{i}$ into an event embedding $E_{i}$, which should ideally exploit the spatiotemporal information from asynchronous events. As a result, we can formulate this mapping as follows:

where the designed kernel function $\mathcal{K}(x,y,t)$ can be deep neural networks or handcrafted operations.

In this work, we attempt to encode the event temporal bin into three existing event representations (i.e., event images [17], voxel grids [75] and sigmoid representation [39]) owning to an accuracy-speed trade-off. Actually, any event representation can be an alternative because our SODFormer provides a generic interface accepting various input types of the event-based object detector.

In DETR [20], the multi-head self-attention (MSA) combines multiple attention heads in parallel to increase the feature diversity. To be concrete, the feature maps are first extracted via CNN-based backbone. Then, $X_{q}$ and $X_{k}$ are derived from these feature maps as query and key-value in attention mechanism with both sizes of $(HW,d)$. Let $q$ refer to a query element with the feature map $X_{q}$, and $k$ indexes a key-value element with the feature map $X_{k}$. The MSA operation integrates the outputs of $M$ attention heads as:

where $m$ indexes the single attention head, $\bm{W}_{m}$ and $\bm{W}_{m}^{\prime}$ are learnable weights. $A_{mqk}=\exp(\frac{\bm{X}_{q}^{T}\bm{W}_{mq}^{T}\bm{W}_{mk}\bm{X}_{k}}{\sqrt{d_{k}}})$ is the attention weight, in which $\bm{W}_{mq}$ and $\bm{W}_{mk}$ are also learnable weights, $d_{k}$ is a scaling factor.

To achieve fast convergence and high computation efficiency, Deformable DETR [21] designs a deformable multi-head self-attention (DMSA) operation to attend the local $L$ sampling points instead of all pixels in feature map $X$. Specifically, DMSA first determines the corresponding locations of each query in other feature maps, which we refer to as reference points, and then adds the learnable sampling offsets to reference points to obtain the locations of sampling points. It can be described as:

where $P_{q}$ is a 2D reference point, $\Delta P_{mlq}$ denotes the sampling offset relative to $P_{q}$, and $A_{mlq}$ is the learnable attention weight of the $l^{th}$ point in $m^{th}$ self-attention head.

In this study, we adopt the encoder of the Deformable DETR as our spatial Transformer encoder (STE) to extract features from each visual stream. Meanwhile, we further extend the spatial DMSA strategy to the temporal domain (Section 5.4), which leverages rich temporal cues from continuous event stream and adjacent frames.

The temporal Transformer aggregates multiple spatial feature maps from the spatial Transformer and generates the predicted bounding boxes. It includes two main modules: temporal deformable Transformer encoder (TDTE) (Section 5.4.1) and temporal deformable Transformer decoder (TDTD) (Section 5.4.2).

To integrate two heterogeneous visual streams, the most existing fusion strategies [49, 13, 14, 15, 16] usually first split asynchronous events into discrete temporal bins synchronized with frames, and then fuse two streams via the post-processing or feature aggregation (e.g., averaging and concatenation). However, these synchronized fusion strategies may have two limitations: (i) The post-processing or feature aggregation operations cannot distinguish the degradation degree of different modalities and regions, which means they are difficult to eliminate the unimodal degradation thoroughly and thus incur the bottleneck of performance improvement; (ii) The joint output frequency may be limited by the sampling rate of conventional frames (e.g., 25 Hz), which fails to meet the needs of fast object detection in real-time high-speed scenarios. Therefore, we design an asynchronous attention-based fusion module to overcome the two limitations mentioned above.

One key question of two heterogeneous visual streams is how to achieve the high joint output frequency on demand for a given task. In general, the high temporal resolution event stream can be split into discrete event temporal bins, the frequency of which can be flexibly adjusted to be much larger than the conventional frame rate in real-time high-speed scenarios. As shown in Fig. 7, the event flow has more temporal sampling timestamps than the frame flow in an asynchronous manner. For the timestamp $t_{i}$, $X_{S}$ is the output of the FFN in the TDTE. The proposed fusion strategy first searches the most recent sampling timestamp $t_{j}$ in the frame flow, which can be described as:

where $\Delta t=t_{i}-t_{k}$ is the temporal difference between the event sampling timestamp and the frame sampling timestamp.

Another key question of two complementary data is how to tackle the unimodal degradation that the feature aggregation operations (e.g., averaging) fail to overcome and implement pixel-wise adaptive fusion for better performance. Unlike the previous weight mask by a multi-layer perceptron (MLP) [78], we compute the pixel-wise weight mask using the attention mechanism, which simplifies our model and endows it with stronger interpretability. Our fusion strategy starts with two intuitive assumptions: (i) the stronger the association between two pixels is, the greater the attention weight between them will be, and (ii) the pixels in degraded regions are supposed to have a weaker association with other pixels. Combining the two points above, we conclude that the sum of attention weights between a pixel and all other pixels represents the degradation degree of that pixel to some extent, and thus can serve as the weight for that pixel during fusion. Next, we take event flow as an example to illustrate how our attention weight mask module (see Fig. 5) obtains fusion weights. To implement this, we first remove the softmax operation in the standard self-attention to calculate the attention weights $\bm{W}_{XY}^{S}\in\mathbb{R}^{HW\times HW}$ between each pixel, and accumulate attention weights for each pixel $x$ as $\bm{W}_{x}^{S}=\sum_{y}\bm{W}_{xy}^{S}$, which can be summarized as follows:

where $\bm{W}_{X}^{S}\in\mathbb{R}^{WH\times 1}$ is the weight mask with the width $W$ and the height $H$ for event flow. $\bm{M}^{Q_{S}}$ and $\bm{M}^{K_{S}}$ are learnable matrix parameters. $d_{k}$ is a scaling factor in the self-attention [22]. As shown in Fig. 7, the aforementioned steps are exactly the same for the process of frame flow.

Then, we concatenate the weight masks along the pixel dimension and adopt the softmax to that dimension (i.e., softmax of [$\bm{W}_{j}^{I}$, $\bm{W}_{j}^{S}$], $j=1,2,\cdots,HW$) to normalize the pixel-wise attention weights at each pixel. Meanwhile, we further split the weight masks along the pixel dimension to obtain the frame weight masks $\bm{M}_{I}$ and the event weight masks $\bm{M}_{S}$ as follows:

Finally, we can obtain the fused feature map $F_{f}$ by summing up the unimodal feature maps with the corresponding pixel-wise weight masks by:

where $\oplus$ denotes the sum operation, and $\odot$ refers to the element-wise multiply operation.

In this part, we will present the dataset, implementation details, and evaluation metrics as follows.

Dataset and Setup. Our PKU-DAVIS-SOD dataset is designed for streaming object detection in driving scenes, which provides two heterogeneous and temporally long-term visual streams with manual labels at 25 Hz (see Section 3). This newly built dataset contains three subsets including 671.3k labels for training, 194.7k for validation, and 214.1k for testing. Each subset is further split into three typical scenarios (i.e., normal, low-light, and motion blur).

Implementation Details. We select event images [17] as the event representation and the ResNet50 [74] as the backbone to achieve a trade-off between accuracy and speed. All parameters of the backbone and the spatial Transformer encoder (STE) are shared among different temporal bins of the same modality. We set the attention head $M$ to 8 and the sampling point $L$ to 4 for the deformable multi-head self-attention (DMSA) in Eq.(7) and the temporal deformable multi-head self-attention (TDMSA) in Eq.(8). The temporal aggregation size in TDMSA is set to 9 owning to the balance of the accuracy and the speed. During training, the height of an image is randomly resized to a number from [256: 576: 32], and the width is obtained accordingly while maintaining the aspect ratio. Similarly, for evaluation, all images are resized to 352 in height and 468 in width to maintain the aspect ratio. Other data augmentation methods such as crop and horizontal flip are not utilized because our temporal Transformer requires accurate reference points. Following the Deformable DETR [21], we adopt the matching cost and the Hungarian loss for training with loss weights of 2, 5, 2 for classification, $L_{1}$ and GIoU, respectively. All networks are trained for 25 epochs using the Adam optimizer [79], which is set with the initial learning rate of $2\times 10^{-4}$, the decayed factor of 0.1 after the 20th epoch, and the weight decay of $10^{-4}$. All experiments are conducted on NVIDIA Tesla V100 GPUs.

Evaluation Metrics. To compare different approaches, the mean average precision (e.g., COCO mAP [80]) and running time ($ms$) are selected as two evaluation metrics, which are the most broadly utilized in the object detection task. In the effective test, we give a comprehensive evaluation using average precision with various IoUs (e.g., AP, AP_0.5, and AP_0.75) and AP across different scales (i.e., AP${}_{\text{S}}$, AP${}_{\text{M}}$, and AP${}_{\text{L}}$). In other tests, we report the detection performance using the AP_0.5. Following the video object detection, we compute the AP for each class and the mAP for three classes in all labeled timestamps. In other words, we evaluate the object detection performance with the output frequency of 25 Hz in every labeled timestamp.

The objective of this part is to assess the validity of our SODFormer, so we implement two main experiments including performance evaluation in various scenarios (Section 6.2.1) and comparison with state-of-the-art methods (Section 6.2.2) as follows.

In this section, we implement an ablation study to investigate how parameter setting and each design choice influence our SODFormer.

This subsection will present the predicted output slot analysis (Section 6.4.1) and the visualization of the designed temporal deformable attention (Section 6.4.2). Then, we further present how to process two heterogeneous visual streams in an asynchronous manner (Section 6.4.3). Finally, we analyze some failure cases of our SODFormer (Section 6.4.4).