GelSplitter: Tactile Reconstruction from Near Infrared and Visible Images

GelSight [28] is a well-established VT sensor that operates like a pliable mirror, transforming physical contact or pressure distribution on its reflective surface into tactile images that can be captured by a single camera. Building upon the framework of the GelSight, various VT sensors [9, 22, 23] are designed to meet diverse application requirements, such as  [18, 16, 17, 10]. Inspired by GelSight, DIGIT [13] improves upon past the VT sensors by miniaturizing the form factor to be mountable on multi-fingered hands.

Additionally, there are studies that explore the materials and patterns of reflective surfaces to gather other modal information of touch sensing. For example, FingerVision [27] provides a multi-modal sensation with a completely transparent skin. HaptiTemp [2] uses thermochromic pigments color blue, orange, and black with a threshold of 31^∘C, 43^∘C, and 50^∘C, respectively on the gel material, to enable high-resolution temperature measurements. DelTact [31] adopts an improved dense random color pattern to achieve high accuracy of contact deformation tracking.

The VT sensors mentioned above are based on a single RGB camera. Inspired by binocular stereo vision, GelStereo [6] uses two RGB cameras to achieve tactile geometry measurements. Further, the gelstereo is developed in six-axis force/torque estimation [30] and geometry measurement [7, 11].

In current research, one type of camera is commonly used, with a small size and a simplified optical path. The RGB camera solution for binocular stereo vision effectively estimates disparity by triangulation. However, data alignment is a common issue to different kinds of camera images [3], because multi-modal sensors naturally have different extrinsic parameters between modalities, such as lens parameters and relative position. In this paper, two identical imaging windows are fabricated by a splitting prism, which filter RGB component (bandpass filtering at 650$nm$ wavelength) and IR component (narrowband filtering at 940$nm$ wavelength).

Multi-modal image fusion is a fundamental task for robot perception, healthcare and autonomous driving [26, 21]. However, due to high raw data noise, low information utilisation and unaligned multi-modal sensors, it is challenging to achieve a good performance. In these applications, different types of datasets are captured by different sensors such as infrared (IR) and RGB image [32], computed tomography (CT) and positron emission tomograph (PET) scan [19], LIDAR point cloud and RGB image [15].

Typically, the fusion of NIR and RGB images enhances the image performance and complements the missing information in the RGB image. DenseFuse [14] proposes an encoding network that is combined with convolutional layers, a fusion layer, and a dense block to get more useful features from source images. DarkVisionNet [20] extracts clear structure details in deep multiscale feature space rather than raw input space by a deep structure. MIRNet [29] adopts a multi-scale residual block to learn an enriched set of features that combines contextual information from multiple scales, while simultaneously preserving the high-resolution spatial details.

Data alignment is another important content of multi-modal image fusion. Generally, the targets corresponding to multiple sensors are in different coordinate systems, and the data rates of different sensors are diverse. To effectively utilize heterogeneous information and obtain simultaneous target information, it is essential to map the data onto a unified coordinate system with proper time-space alignment [24, 21].

Following the above literature, PFSNN is designed for the gelsplitter, completing the missing information of RGB images by NIR image. It generates refined normal vector maps and depth maps. In addition, splitting prisms are embeded to unify the image planes and optical centres of the different cameras and align the multi-modal data.

To capture the NIR image and the RGB image, a splitting prism, band pass filter, narrow band filter and diffuser were prepared . These components ensure that the images are globally illuminated and homogeneous.

The splitting prism is a cube with a side length of 15$mm$, as shown in Fig. 4 (a). It has a spectral ratio of 1:1 and a refractive index of 1.5168, creating two identical viewports. Among the six directions of the cube, except for two directions of the cameras and one direction of the gel, the other three directions are painted with black to reduce secondary reflections.

The diffuser in lighting can produce more even global illumination as it distributes light evenly throughout the scene. The octagonal diffuser needs to be optically coupled to the splitting prism to avoid reflection from the air interface, as shown in Fig. 4 (b). “3M Diffuser 3635-70” is used for the sensor.

A 650$nm$ bandpass filter and 940$nm$ narrowband filter are used to separate the RGB component from the NIR component. The lens and filter are integrated together, as shown in Fig. 4 (c).

In this paper, five types of colour LEDs are provided for illumination in different directions, including red ( Type NCD0603R1, wavelength 615$\sim$630$nm$), green ( Type NCD0603W1, wavelength 515$\sim$530$nm$), blue ( Type NCD0603B1, wavelength 463$\sim$475$nm$), white ( Type NCD0603W1) and infrared ( Type XL-1608IRC940, wavelength 940$nm$), as shown in Fig. 4 (d). These LEDs are all in 0603 package size (1.6×0.8 $mm$).

Moreover, in order to represent surface gradient information, LEDs are arranged in two different ways. Firstly, red, green, blue and white LEDs can be arranged in rows, illuminating the four sides of the gel. This allows for a clear representation of surface gradients from different angles. Secondly, infrared LEDs can be arranged in a circular formation above the gel, surrounding the infrared camera. This allows IR light to shine on the shaded areas, supplementing the missing gradient information.

Finally, we designed the FPC and PCB to drive LEDs, as shown in Fig. 4 (e). The LED brightness is configured with a resistor, and its consistency is ensured by a luminance meter.

Both NIR and RGB cameras are used the common CMOS sensor OV5640, manufactured by OmniVision Technologies, Inc. The OV5640 supports a wide range of resolution configurations from 320×240 to 2592×1944, as well as auto exposure control (AEC) and auto white balance (AWB). Following the experimental setting of GelSights [1], our resolution is configured to 640×480. AEC and AWE are disabled to obtain the linear response characteristics of the images.

To capture clear images, it is essential to adjust the focal length by rotating the lens and ensure that the depth of field is within the appropriate range. Although ideally, the optical centres of the two cameras are identical. In practice, however, there is a small assembly error that requires fine-grained data alignment.

Both RGB and NIR cameras are calibrated and corrected for aberrations using the Zhang’s calibration method implemented in OpenCV. Random sample consistency (RANSAC) regression is used to align the checkerboard corners of the two cameras’ images to achieve data alignment.

Our transparent silicone is coated with diffuse reflective paint. We choose low-cost food grade platinum silicone which operates in a 1:1 ratio. The silicone is poured into a mould and produced as a soft gel mat with a thickness of 1.5 $mm$. In our experiment, we found that a gel with a Shore hardness of 10A is both pliable enough to avoid breakage and capable of producing observable deformations. To achieve defoaming, it is necessary to maintain an environmental temperature of approximately 10°C and apply a vacuum pressure of -0.08 MPa.

We use stereo lithography appearance (SLA) 3D printing to create the case. Compared to fused deposition modelling (FDM) 3D printing of PLA, SLA technology has a higher precision for assemble requirements of the splitting prism. Despite the addition of a prism and a NIR camera, our case still maintains a comparable size to the state-of-the-art VT sensor, GelSlim 3.0 [22], as shown in Fig.  4 (g).

The process of 3D tactile reconstruction can be divided into two main steps. The first step involves calculating the normal map from RGB-NIR. The second step is to apply the fast Poisson solver [28] to integrate the gradients and obtain the depth information. This helps to create a detailed 3D tactile reconstruction, which can be very useful for various applications.

The screw cap, screw, hair, fingerprint is chosen to qualitative testing of resolution and precision, as shown in Fig. 6. In NIR images, the opposite property to RGB images is observed, with higher depth map gradients being darker. This means that our design allows the NIR image to contain information that is complementary to the RGB image and illuminates the shaded parts of the RGB image,as shown in Fig. 6 (a). In addition, our splitting design of imaging allows both the normal vector map and the reconstructed depth map to have a clear texture, as shown in Fig. 6 (b), where the threads of the screw are clearly reconstructed. Hair strands (diameter approx. 0.05$mm$ $\sim$ 0.1$mm$) and fingerprints(diameter approx. 0.01$mm$$\sim$0.02$mm$) are used to test the minimum resolution of the GelSplitter, as shown in Fig. 6 (c) (d).

In addition, our splitter can be easily switched between RGB and RGB-NIR modes and provides a fair comparison of the results, as shown in the Tab. 2. The LUT method [28] is employed as a baseline to verify the validity of our method. The results show that the addition of NIR reduces the normal error by 0.561° and 0.375° for LUT and PSFNN respectively. Our PFSNN outperforms the LUT, decreasing the error by over 40%.