Listen to Your Map:
An Online Representation for Spatial Sonification

We employ a modified VINS-RGBD [4] localisation framework to generate precise camera poses as inputs for the mapping. A key aspect of our approach is the improved synchronisation between the estimated poses and the raw dense point cloud data captured by the RGB-D camera. By refining the VINS framework, we ensure consistent spatial alignment, which enhances the accuracy and reliability of the mapping process, ultimately enabling more precise and detailed scene reconstruction.

Our representation is based on the VDB-GPDF mapping framework as proposed in [1]. It couples the VDB grid structure with the Gaussian Process Distance Field [19, 20, 21] to gradually construct and maintain a large-scale, dense distance field map of the environment. The depth sensor data (from depth cameras or LiDARs) at the world coordinate frame is voxelised to create a local VDB structure with 3D points. In every VDB leaf node, the local voxel centres act as training points for GPs. The collection of GPs across all leaf nodes forms the temporary latent Local GP Signed Distance Field (L-GPDF).

For testing, a set of query points is generated by ray-casting from the sensor origin through the voxels in the current reference frame. Each test point is used to query the L-GPDF for the distance field and variance estimates. The queried values from L-GPDF are then fused into a global VDB grid map by updating the voxel distances using the weighted sum method. Following the fusion of the distances and surface properties, the marching cubes algorithm [22] is used to reconstruct a dense surface by identifying the active voxels in the global VDB. The Global Gaussian Process Distance Field (G-GPDF) is made up of the zero-crossing points from active leaf nodes. At this stage, queries for the G-GPDF are computed by averaging the inferred distances from neighbouring GP nodes. For further details refer to [1].

Spatial audio feedback requires accurate but simple information, thus we propose to represent the incrementally built dense 3D map of the environment as a 2D circular sensor-centric representation. To ensure our representation aligns with human navigation requirements, we segment the ground and non-ground points. Any point above the ground and in the height range of a human that can potentially be an obstacle is classified as a non-ground point. Then, we project all these 3D non-ground points onto a 2D plane. On this plane, every 3D point is reduced to its corresponding 2D coordinates, maintaining spatial relationships within the scene.

A circular grid centred at the location of the depth sensor is created by rasterising these 2D points. Rasterisation is faster and more efficient than raycasting, making it ideal for real-time processes and handling complex scenes. We do this by calculating the radial distance and azimuthal angle for every point with respect to the sensor. The azimuthal angle is derived by calculating the cross-product between the sensor’s directional vector and the vector extending from the sensor to the point of interest.

After that, this angle is normalised to fit inside a 360° circular representation. Each degree denotes a distinct angle surrounding the position of the sensor. The distance between the point and the sensor is used to calculate the radial distance. This enables us to rasterise each point, according to its distance from the sensor, onto a circular plane.

This process of rasterisation yields a complete 360° depiction of the scene, with every point corresponding to a distinct angle and distance from the sensor. This ensures capturing the closest visible points surrounding the sensor’s position. Sonifying the three-dimensional scene is especially advantageous with this minimalist circular representation. As we explain in the following section, each point in the circle can be associated with a specific sound, allowing for intuitive sound cues that represent the 3D environment. We ensure that the most pertinent spatial information is preserved by keeping only the closest points at each angle, which improves the audio feedback’s clarity.

We take our representation further onto generating a 3D cylindrical structure instead of only a circle. This approach is guided by the intuition that the human is positioned within the cylinder. The reconstructed 3D mesh is rasterised onto a cylindrical surface to create the 3D cylindrical representation. This representation provides a structured, sensor-centric view of the surroundings. The cylindrical grid is discretised into vertical (elevation) and azimuthal intervals. The elevation spans heights between 0.1 and 2 meters above the sensor, while the azimuth is divided into 360-degree intervals.

Each 3D point in the scene is mapped to a specific azimuth-elevation pair, and only the closest point (in terms of radial distance) for each pair is retained. This process effectively reprojects the 3D points onto a simplified 2D representation of the scene. The reprojection occurs by collapsing the radial dimension, projecting all points onto a central 2D plane or line located at the center of the cylinder. The resulting 2D map provides a compact representation of the scene’s geometry, where each point in the azimuth-elevation grid corresponds to the nearest surface point along the cylinder’s radial direction.

We demonstrate the ability of our proposed representation to provide spatial information for cognitive sonification. An auditory framework is applied with the spatial representation to generate binaural auditory signals. A foundational aspect of the sonification process is to leverage the natural auditory perspective provided by human spatial hearing. In other words, humans naturally perceive the location of sounds around them and we design the representation to take advantage of spatial hearing perception.

In this work, augmented or virtual spatial audio is enabled via the use of binaural room impulse responses (BRIRs) which are filters that mathematically specify the transformation of sound from a particular source location in a room to the ears of the individual listening to the map. There is a separate BRIR filter for each ear and each source location. Note that BRIR filters are unique for each individual in a perceptually relevant way because of morphological differences in ear shape. The data accumulated from the map representation has the spatial geometry of the room or environment. This information is sufficient to compute BRIRs using room simulation techniques. However, in this work, we use custom BRIRs that were prerecorded in a representative laboratory environment that includes a polypropylene carpet, a motion capture metallic frame, wooden beams with speakers mounted on them and other furniture (see Fig. 2).

The BRIR measurements were recorded using the Brüel & Kjær type 4128C Head and Torso Simulator (HATS) with two in-ear microphones together with a Brüel & Kjær type 9640 turntable. We arranged 10 small loudspeakers in a row at a height consistent with the audiovisual horizon of the HATS model. The 10 loudspeakers were positioned directly in front of the HATS model at zero degree of azimuth at distances varying from 0.4 m to 4 m in steps of 0.4 m. Using the turntable, the HATS model was rotated and BRIRs were recorded every $3^{\circ}$ of azimuth from $-177^{\circ}$ to $180^{\circ}$. The BRIR files were saved as Spatially Oriented Format for Acoustics (SOFA) files [23]. The full loudspeaker recording positions are shown in Fig. 3.

In order to demonstrate the suitability of the mapping representation for spatial sonification, we focus on the 2D circular representation. We sonify the distance to the surface for all directions around the circle counter-clockwise. It simulates a scanning “length-changing cane”. The cane automatically taps the closest surface in each discrete direction as it scans. As we mentioned, the BRIRs were recorded every $3^{\circ}$. Note that, our circular and cylindrical representations have $1^{\circ}$ resolution. To avoid sensory overload, we choose $10^{\circ}$ as the resolution to separate the 360-degree range data into 36 angular sectors.

For each angular sector, the angle and range corresponding to the smallest range value are selected. The selected range data for each sector is then converted to binaural spatial audio by applying the appropriate BRIR filter to a ‘tap’ sound. Note that if a particular angular sector has not yet been scanned, a ‘woosh’ sound is played instead of a ‘tap’ sound. The distances encompassing the range data are scaled into the 0.4 m to 4 m range of the BRIRs using a constant scaling chosen appropriately for the given environment. Distances beyond 4 m are clamped to 4 m in the BRIRs range. This sonification distance range fits well with the perceptual properties of human distance perception, which is significantly poorer than human auditory direction perception [24] and we generally perceive distance in the near-field ($<$ 2 meters) better than the far-field ($>$ 2 meters). To further enhance the distance perception we apply a slight pitch shift to the tap sound. ‘Tap’ sounds closer than 1.5 m are shifted down in pitch by four semitones and ‘tap’ sounds further than 2.5 m are shifted up in pitch by four semitones. Our accompanying video shows these effects. Listening through headphones is recommended to hear the sounds at the correct azimuth location.

We first evaluate the time consumption to show our representation can provide online performance for sonification. Fig. 4 shows the statistical computational time for the Cow and Lady dataset. Note that our framework has the free-space carving methods [1] enabled to update the moving objects in the scene. For the sake of simplicity, we compute the time for all the processes, including fusion, 2D circle, and 3D cylinder. It illustrates that our representation is efficient with varying resolutions for online performance.

For the accuracy evaluation, we use the raw depth images in the same way as in our framework to compute the 2D circle and 3D cylinder. Note that we apply the ground truth point cloud of the Cow and Lady to calculate the ground truth circle and cylinder. We then quantitatively compare our circle and cylinder representations and depth images with different map resolutions against the ground truth. As demonstrated in Fig. 5, at the beginning of the dataset, all RMSEs are high due to noisy measurements when the drone is taking off. Then the RMSE drops due to more information is collecting along the steady exploration. The RMSE of the depth sensor circle and cylinder is noisy due to the lack of fusion over time. Our accuracy outperforms depth representations in different resolutions. Note that occasionally, the depth sensor RMSE has lower values than ours, and this is because the depth sensor its only calculated in the field of view of the camera. Therefore, compared to GT and ours, only a few points in the depth circle and cylinder have information and are being evaluated for RMSE.

We compare the coverage of our framework against the depth sensor again. Due to the ground truth having missing areas, we use the fully reconstructed mesh as the benchmark to compute the coverage. Demonstrating in Fig. 6, as the mapping voxel resolution varies from 5cm to 15cm, the coverage of the depth representations remains the same. The depth circle covers below 20%, and the cylinder covers around 10%. Our incremental representation accumulates and fuses measurements as the coverage grows over frames. Note that the coverage gets up and down occasionally due to passing through unexplored areas to gather more information. With a high resolution of 5cm, our circle reaches 100% coverage, and our cylinder covers over 90% at the end.

We aim to show that our representation has the ability to deal with dynamic objects, which is very common in the real-world scenario. Our framework takes poses and points as raw input, so we use localisation frameworks such as VINS [4] to provide proper poses. We move a live camera to map the scene online. In Fig. 7, our 2D circle is fully reconstructed in 360 degrees, and we can visual object projection on the circle for each angle clearly. From Fig. LABEL:dynamic_1 to Fig. LABEL:dynamic_4, there was an object in the scene and later moved away. Our circle is updated with respect to the latest status of the scene.

This section aims to show the suitability of our representation for sonification over the Euclidean distance fields (EDF), which is better suited for path planning and navigation. For the EDF [1], it usually has a distance value and the gradient of the distance given the point’s location in the space. It always measures the distance and direction from the point to the closest surface. As we show in Fig. LABEL:arrow_ours, our circle has bearing angles pointing 360 degrees, and for each bearing, we have the closest distance along the ray (the ray arrow is shown in blue). However, as we see from Fig. LABEL:arrow_esdf, we query the EDF for each point on the circle, and this gives us the distance and gradient to the closest surfaces. The gradients of distances (shown in blue) on the circle slip around as the closest surface could be in any direction. This is not suitable for sonification as the bearing does not vary sequentially.

In our accompanying video and Fig. 9, we show an incrementally built and sonified room-sized environment using the proposed framework. Incremental localisation (using VINS [4]), mapping (with VDB-GPDF [1]), circular rasterization and sonification are shown in real-time. Listening through headphones is recommended to hear the sounds at the correct azimuth location.