Perspective Phase Angle Model for
Polarimetric 3D Reconstruction

This topic is closely related to two lines of research: shape from polarization (SfP) and photo-polarimetric stereo. SfP first estimates surface normals that are parameterized in the OPA model by azimuth and zenith angles from polarization phase angles and degree of linear polarization and then obtains Cartesian height maps by integrating the normal maps [17, 2]. With additional spectral cues, refractive distortion [12] of SfP can be solved [11]. Alternatively, it has been shown that the surface normals, refractive indexes, and light directions can be estimated from photometric and polarimetric constraints [18]. The recent work of DeepSfP [4] is the first attempt to use a convolutional neural network (CNN) to estimate normal maps from polarization images. It is reasonable to expect more accurate surface normal estimation when these methods parameterize the surface normal with the help of the proposed PPA model in perspective cameras. In fact, a recent work shows that learning-based SfP can benefit from considering the perspective effect [14]. Regardless, as a two-stage method, SfP is sensitive to noise.

As a one-stage method, photo-polarimetric stereo directly estimates a height map from polarization images and is able to avoid cumulative errors and suppress noise. By constructing linear constraints on surface heights from illumination and polarization, height map estimation is solved through the optimization of a non-convex cost function [22]. Yu et al. [31] derives a fully differentiable method that is able to optimize a height map through non-linear least squares optimization. In [26], variations of the photo-polarimetric stereo method are unified as a framework incorporating different optional photometric and polarimetric constraints. In these works, the OPA constraint is a key to exploiting polarization during height map estimation. In perspective cameras, the proposed PPA constraint can provide a more accurate description than the OPA constraint.

Surface reconstruction can be solved by optimizing a set of functionals given phase angle maps of three different views [20]. To address transparent objects, a two-view method [17] exploits phase angles and degree of linear polarization to solve correspondences and polarization ambiguity. Another two-view method [3] uses both polarimetric and photometric cues to estimate reflectance functions and reconstruct shapes of practical and complex objects. By combining space carving and normal estimation, [16] is able to solve polarization ambiguity problems and obtain more accurate reconstructions of black objects than pure space carving. Fukao et al. [10] models polarized reflection of mesoscopic surfaces and proposes polarimetric normal stereo for estimating normals of mesoscopic surfaces whose polarization depends on illumination (i.e., polarization by light [6]). These works are all based on the OPA model without considering the perspective effect.

Recently, traditional multi-view 3D reconstruction methods enhanced by polarization prove to be able to densely reconstruct textureless and non-Lambertian surfaces under uncalibrated illumination [7, 30, 21]. Cui et al. [7] proposes polarimetric multi-view stereo to handle real-world objects with mixed polarization and solve polarization ambiguity problems. Yang et al. [30] proposes a polarimetric monocular dense SLAM system that propagates sparse depths in textureless scenes in parallel. Shakeri et al. [21] uses relative depth maps generated by a CNN to solve the $\pi/2$-ambiguity problem robustly and efficiently in polarimetric dense map reconstruction. In these works, iso-depth contour tracing [33] is a common step to propagate sparse depths on textureless or specular surfaces. It is based on the proposition that, with the OPA model, the direction perpendicular to the phase angle is the tangent direction of an iso-depth contour [7]. However, in a perspective camera, this proposition is only an approximation.

Besides, the OPA constraint can be integrated in stereo matching [5], depth optimization [30] and mesh refinement [32]. However, only the first two components of the surface normal are involved in the constraint, in addition to its inaccuracy in perspective cameras. With the proposed PPA model, all three components of a surface normal are constrained and the constraint is theoretically accurate for a perspective camera.

The polarization phase angle of light emitted from a surface point depends on the camera pose. Therefore, it is intuitive to use this cue for camera pose estimation. Chen et al. [6] proposes polarimetric three-view geometry that connects the phase angle and three-view geometry and theoretically requires six triplets of corresponding 2D points to determine the three rotations between the views. Different from using only phase angles in [6], Cui et al. [8] exploits full polarization information including degree of linear polarization to estimate relative poses so that only two 2D-point correspondences are needed. The method achieves competitive accuracy with the traditional five 2D-point method. Since both works are developed without considering the perspective effect, the proposed PPA model can be used to generalize them to a perspective camera.

As a function of the orientation of the polarizer $\phi$, the intensity of unpolarized light is attenuated sinusoidally as $I(\phi)=I_{avg}+\rho I_{avg}\cos(2(\phi-\varphi))$. The parameters of polarization state are the average intensity $I_{avg}$, the degree of linear polarization (DoLP) $\rho$ and the phase angle $\varphi$. The phase angle is also called the angle of linear polarization (AoLP) in [32, 28, 24, 21]. In this paper, we uniformly use the term “phase angle” to refer to AoLP.

Given images captured through a polarizer at a minimum of three different orientations, the polarization state can be estimated by solving a linear system [30]. For a DoFP polarization camera that captures four images $I(0)$, $I(\frac{\pi}{4})$, $I(\frac{\pi}{2})$ and $I(\frac{3\pi}{4})$ in one shot, the polarization state can be extracted from the Stokes vector $\mathbf{s}=[s_{0},s_{1},s_{2}]^{T}$ as follows:

where $s_{o}=I(0)+I(\frac{\pi}{2})$, $s_{1}=I(0)-I(\frac{\pi}{2})$ and $s_{1}=I(\frac{\pi}{4})-I(\frac{3\pi}{4})$. Although it has been shown that further optimization of the polarization state from multi-channel polarization images is possible [25], for this paper, Eq. (1) is sufficiently accurate for us to generate ground-truth phase angle maps from polarization images.

The phase angles estimated from polarization images through Eq. (1) are directly related to the surface normals off which light is reflected. It is this property that is exploited in polarimetric 3D reconstruction research to estimate surface normal from polarization images. Model of projection, which defines how light enters the camera, is another critical consideration in order to establish the relationship between the phase angle and the surface normal.

The OPA model assumes that all light rays enter the camera in parallel, as shown by the blue plane in Fig. 2, so that the azimuth angle of the surface normal $\mathbf{n}$ is equivalent to the phase angle $\varphi_{o}$ up to a $\pi/2$-ambiguity and a $\pi$-ambiguity [32]. Specifically, $\mathbf{n}$ can be parameterized by the phase angle $\varphi_{o}$ and the zenith angle $\theta$ in the camera coordinate system:

With this model, $\varphi_{o}$ can be calculated from $\mathbf{n}$ as

We can use Eq. (3) to evaluate the accuracy of the OPA model. I.e., if the OPA model is accurate, then the phase angle calculated by Eq. (3) should agree with that estimated from the polarization images. Although this model is based on the orthographic projection assumption, it is widely adopted in polarimetric 3D reconstruction methods even when polarization images are captured by a camera with perspective projection. In addition, Eq. (2) defines a constraint on $\mathbf{n}$ by $\varphi_{o}$, i.e., the OPA constraint:

This linear constraint is commonly integrated in depth map optimization [7, 30, 26]. It is also the basis of iso-depth contour tracing [1, 7] as it gives the tangent direction of an iso-depth contour. Obviously, this constraint is strictly valid only in the case of orthographic projection.

The geometric properties of a polarizer in [13] inspire us to model the phase angle as the direction of the intersecting line of the image plane and the PoI spanned by the light ray and the surface normal. As shown in Fig. 2, there is an obvious difference between the two definitions of the phase angle, depending upon the assumed type of projection.

Specifically, let the optical axis be $\mathbf{z}=[0,0,1]^{T}$ and the light ray be $\mathbf{v}=[v_{x},v_{y},v_{z}]^{T}=K^{-1}\mathbf{x}/\|K^{-1}\mathbf{x}\|$, for an image point at the pixel coordinates $\mathbf{x}=[u,v,1]^{T}$ and a camera with intrinsic matrix $K$. From Fig. 2, the PPA model can be formulated as follows:

where $\mathbf{z}$, $\mathbf{v}$ and $\mathbf{n}$ are normalized to 1, $c$ is a constant and $\mathbf{d}=[\cos\varphi_{p},-\sin\varphi_{p},0]^{T}$. From Eq. (5), the phase angle $\varphi_{p}$ can be obtained from $\mathbf{n}$ and $\mathbf{v}$ as:

Similar to Eq. (3), Eq. (6) can also be used for evaluating the accuracy of the PPA model. Different from $\varphi_{o}$, $\varphi_{p}$ not only depends on the surface normal $\mathbf{n}$ but also on the light ray $\mathbf{v}$.

In addition, $\mathbf{n}$ can be parameterized by $\varphi_{p}$ and $\theta$ as $\mathbf{n}=-(e^{\theta\mathbf{a}^{\land}_{p}})\mathbf{v}$ where $\mathbf{a}_{p}=\mathbf{d}\times\mathbf{v}/\|\mathbf{d}\times\mathbf{v}\|$ is the normal of the PoI and $\mathbf{r}^{\land}$ represents the skew-symmetric matrix of $\mathbf{r}\in\mathbb{R}^{3}$. Eq. (2) can also be reformulated into a similar form as $\mathbf{n}=-(e^{\theta\mathbf{a}^{\land}_{o}})\mathbf{v}$ where $\mathbf{a}_{o}=[-\sin\varphi_{o},\cos\varphi_{o},0]^{T}$. Obviously, the difference of the two parameterizations of $\mathbf{n}$ is the axis about which $\mathbf{v}$ rotates.

With the PPA model, the PPA constraint can be derived from Eq. (5) as follows:

Note that Eq. (7) can be used just as the OPA constraint Eq. (4), without the knowledge of object geometry since $\mathbf{v}$ is a scene-independent directional vector. Compared with Eq. (4), Eq. (7) has an additional constraint on the third component of the normal. This additional constraint is critical in allowing us to estimate $\mathbf{n}$ of a planar surface from a single view as we will show in the next section. Table 1 summaries the definitions of the phase angle, the normal and the constraint of the OPA and PPA models.

Comparing the two constraints in Table 1, the coefficient $-(v_{y}\cos\varphi_{p}+v_{x}\sin\varphi_{p})/v_{z}$ implies that there exist four cases in which the two models are equivalent:

1. 1.

   $\mathbf{v}=[0,0,1]^{T}$, when the light ray is parallel to the optical axis, which corresponds to orthographic projection.

2. 2.

   $[v_{x},v_{y}]^{T}\parallel[\cos\varphi_{p},-\sin\varphi_{p}]^{T}\parallel[n_{x},n_{y}]^{T}$, when the PoI is perpendicular to the image plane.

3. 3.

   $n_{z}\to 1$, when the normal tends to be parallel to the optical axis. In this case, the PoI is also perpendicular to the image plane.

4. 4.

   $n_{z}=0$, when the normal is perpendicular to the optical axis.

Besides, according to Eq. (3) and Eq. (4), if $n_{z}=1$, the OPA model and the OPA constraint will become degenerate whereas ours will still be useful.

Theoretically, a surface normal can be solved from at least its two observations with the PPA constraint. The observations can either be the phase angles from multiple pixels in the same phase angle map or from phase angle maps of multiple views as shown in Fig. 3. The two cases lead to the following two methods.

An image capture setup is shown in Fig. 4. Our camera has a lens with a focal length of 6 mm and a field of view of approximately $86.6^{\circ}$. Perspective projection is therefore appropriate to model its geometry. We perform camera calibration first to obtain its intrinsics, and use the distortion coefficients to undistort the images before they are used in our experiments. The calibration matrix $K$ is used to generate the light ray $\mathbf{v}$ in homogeneous coordinates.

We capture a glossy and black plastic board with a size of 300 mm by 400 mm on a table by a handheld DoFP polarization camera [9]. The setting is such that: 1) the board is specular-reflection-dominant so that the $\pi/2$ ambiguity problem can be easily solved and 2) the phase angle maps of the board can be reasonably estimated. Although this setting could be perceived as being limited, it is carefully chosen to verify the proposed model accurately and conveniently.

We create a dataset that contains $282$ groups of grayscale polarization images (four images per group) of the board captured at random view points. The camera poses and the ground-truth normal of the board are estimated with the help of AR tags placed on the table. Ground-truth phase angle maps are calculated from polarization images by Eq. (1). To reduce the influence of noise, we only use the pixels with DoLP higher than $0.1$ in the region of the board, and apply Gaussian blur to the images before calculating phase angle maps and DoLP maps. This dataset is used in all the following experiments.

To evaluate the accuracy of the OPA model and the PPA model, we calculate the phase angle of every pixel with Eq. (3) and Eq. (6), respectively, given the ground-truth normals, and compare the results to the estimated phase angle by Eq. (1). The mean and the root mean square error (RMSE) of the phase angle errors of the OPA model are $-0.18^{\circ}$, and $20.90^{\circ}$ respectively, and the ones of our proposed PPA model are $0.35^{\circ}$ and $5.90^{\circ}$ respectively. Although the phase angle error of both the OPA and the PPA models is unbiased with a mean that is close to zero, the RMSE of the phase angle estimated by the PPA model is only $28\%$ of that by the OPA model in relative terms, and small in absolute terms from a practical point of view. As shown in Fig. 1, the PPA model accurately describes the spatial variation of the phase angle while the phase angle maps of the OPA model are spatially uniform and inaccurate, in comparison with the ground truth.

In addition, we plot the error distributions with respect to the viewing angle (the angle between the light ray and the normal), the azimuth difference (the angle between the azimuths of the light ray and the normal) and the zenith angle of the normal. As shown in Fig. 5, the deviations of the PPA error are much smaller than those of the OPA error. It is shown that pixels closer to the edges of the views have larger errors with the OPA model. Additionally, the errors are all close to zeros in the first three equivalent cases stated in Section 4.2. The high density at viewing angles near $40^{\circ}$ in the first column of Fig. 5 is likely a result of the nonuniform distribution of the positions of the board in the images since they correspond to the pixels at the edges of the views.

This experiment is designed to illustrate the influence of the perspective effect on the iso-depth surface contour tracing with the OPA model and to show the accuracy of the proposed PPA model. We sample 20 seed 3D points on the edge of the board and propagate their depths to generate contours of the board. With the OPA model, the iso-depth contour tracing requires only the depths and a single-view phase angle map. However, according to the proposed PPA model, the iso-depth contour tracing is infeasible in perspective cameras. Therefore, we estimate the normals of the points through the method developed in Section 4.3.2 with two views for generating surface contours. To clearly compare the quality of the contours, our method is set to generate 3D contours that have the same 2D projections on the image plane as those by iso-depth contour tracing. The propagation step size is set to $0.5$ pixel, which is the same as that in [7].

The iso-depth contours of the board are expected to be straight in perspective cameras. However, as shown in Fig. 8, the contours generated by iso-depth contour tracing using the OPA model are all curved and lie out of the ground-truth plane, while those generated by our PPA model are well aligned with the ground-truth plane. We also compute the RMSE of the distances between the points on the contours and the plane. The RMSE of the contours using our PPA model is $2.2$ mm, $25\%$ of that using the OPA model which is $9.6$ mm.