Probabilistic Rotation Representation With an Efficiently Computable Bingham Loss Function and Its Application to Pose Estimation${}^{\text{*}}$

4-dimensional rotation representation is widely used. In particular, quaternion is a popular representation. It is utilized such as in PoseCNN [1], PoseNet [10], and 6D-VNet [11]. Another 4-dimensional representation is an axis-rotation representation, introduced such as in MapNet [12]. Although these 4D representation are valid in some cases, it is known that every $d$-dimensional ($d<5$) rotation representation is “discontinuous” (in the sense of [13]), because the 3-dimensional real projective space $\mathbb{R}P^{3}(\cong\mathop{\mathrm{SO}}(3))$ cannot be embedded in $\mathbb{R}^{d}$ unless $d\geq 5$ [14]. Because these representations has some singular points that prevent the network from stable regression, it is preferred that we use the continuous rotation representation in training neural networks.

Researchers have proposed various high-dimension representations. For example, 9D representation was proposed in [15] by using the singular value decomposition (SVD). 10D representation was proposed in [6] using the 4-dimensional symmetric matrix, which can be used for parametrization of Bingham distribution. We will examine other continuous paramterization in Section II-B.

There are several ways to express the rotational ambiguity. In [3, 16], they tried capturing the ambiguity by using multiple quaternions and minimizing the special loss function. In KOSNet [4], they used 3 parameters for describing camera’s rotation (elevation, azimuth, and rotation around optical axis), and employed Gaussian distribution for describing their ambiguity. These two representations, however, are discontinuous as described in Section II-A, since their dimensions are both lesser than 5.

Another approach is to use Bingham distribution. This distribution is easy to parameterize and has been utilized in pose estimation field. It is used for the model distribution of a Bayesian filter to realize online pose estimation [17], for visual self-localization [7], for multiview fusion [8], and for describing the pose ambiguity of objects with symmetric shape [18]. Moreover, there is a continuous parameterization of this distribution, as Peretroukhin et al.[6] gives an example of it. We adopt Bingham distribution as our probabilistic rotation representation.

An NLL loss function is the common choice and has advantages, as described in the Introduction. When computing Bingham loss, the main barrier is the computation of normalizing constant. One solution of this problem is to take the time to pre-compute the table of normalizing constant. For example, Kume et al. [19] use the saddlepoint approximation technique to construct this table. However, even if we use the pre-computed table, we also need to implement a smooth interpolation function as described in [18] to compute a constant missing in the table and a derivative of each constant for backpropagation. Thus, it is desirable to use Bingham’s negative log likelihood (NLL) loss without the pre-computed table.

Instead of using the Bingham NLL loss, another loss function is defined in [6]. They defined QCQP loss to prevent them from suffering this troublesome computations. While this shows moderate performance by calculating uni-modal Bingham loss implicitly, it is difficult to extend to more complicated model such as mixture models [7, 8], because the normalizing constant is required to mix multiple Bingham distributions. In contrast, our Bingham loss can be easily extended to mixture models.

The Bingham distribution [5] is a probability distribution on the unit sphere $\mathbb{S}^{d-1}\subset\mathbb{R}^{d}$ with the property of antipodal symmetry, which is consistent with the quaternion’s property. We set $d=4$ throughout of this paper because we only consider $\mathbb{S}^{3}$. We define Bingham distribution as follows.

where $\bm{q}\in\mathbb{S}^{3},{D}\in{\mathrm{O}}(4),\bm{\lambda}\in\mathbb{R}^{4}$. Here ${\mathrm{O}}(n)$ denotes the $n$-dimensional orthogonal group. Note that it is a Lie group and its Lie algebra ${\mathfrak{o}}(n)$ is formed by $n$-dimensional skew-symmetric matrices.

Here we define $\mathop{\mathrm{diag}}:\mathbb{R}^{m}\to\mathbb{R}^{m\times m}$ as below.

$\mathcal{C}(\bm{\lambda})$ is called a normalizing factor or a normalizing constant of a Bingham distribution $\mathfrak{B}(\bm{q};{D},\bm{\lambda})$. $\mathcal{C}(\bm{\lambda})$ is defined as below:

where $\mathrm{d}_{\mathbb{S}^{3}}(\cdot)$ is the uniform measure on the $\mathbb{S}^{3}$. Note that a normalizing factor depends only on $\bm{\lambda}$. It is easy to check that for any $c\in\mathbb{R}$,

where $\bm{\lambda}+c=(\lambda_{1}+c,\dots,\lambda_{4}+c)$. Therefore, we can set $\bm{\lambda}$ satisfying

by sorting a column of $D$ if necessary. A processed $D$ and a processed $\bm{\lambda}$ are denoted as $D_{\text{shifted}}$, $\bm{\lambda}_{\text{shifted}}$ respectively It follows directly from the Rayleigh’s quotient formula that

where $\bm{q}_{\lambda_{1}}$ is a column vector of $D$ corresponding to the maximum entry of $\bm{\lambda}$. If we sort $\bm{\lambda}$ as (19), $\bm{q}_{\lambda_{1}}$ coincides with the left-most column vector of $D$.

The negative log-likelihood function of the Bingham distribution can be written as follows:

It had been a hard problem to compute $\mathcal{C}(\bm{\lambda})$ until a high efficient computation method was proposed by [9]. Our loss function is implemented mainly based on [9].

The whole procedure is shown in Algorithm 1. Let $r,\omega_{d}$ be real numbers satisfying

We chose $r=2.5,\,\omega_{d}=0.5$ here. Let $c,h,p_{1},p_{2}$ be defined as

where $d$ is any positive number satisfying $d<c$. We chose $d=c/2$ here. $N$ is a positive integer satisfying $N\geq N_{\text{min}}$. One can choose $N_{\text{min}}$ arbitrarily; however, a too small $N_{\text{min}}$ may lead to unstable computation. We chose $N_{\text{min}}=15$ here.

We define a function $w$ parametrized by $p_{1},\,p_{2}$ in (33) as below.

where $\mathop{\mathrm{erfc}}$ is the complementary error function:

Then we define

Now we can calculate the normalizing constant $\mathcal{C}$ as below

for each $i=1,\dots,4$. Although a calculation result of $\mathcal{C}(\bm{\lambda})$ and ${\partial\mathcal{C}}/{\partial\lambda_{i}}(\bm{\lambda})$ should exactly be a real number, one may get a complex number with the very small imaginary part. In our implementation shown in Algorithm 1, we ignore the imaginary part, assuming that it is sufficiently small.

It is noteworthy that if we set the true value of $\mathcal{C}(\bm{\lambda})$ as $\mathcal{C}_{\text{truth}}(\bm{\lambda})$, we get

for a constant $c>0$ independent from $N$ [23]. This means that we can achieve any accuracy if we set a large enough $N$. In this paper, we set $N=200$ in consideration of the computation time.

In this section, we introduce an application of our representation to an existing pose estimator. We use PoseCNN [1] as the backborn framework. Our implementation is based on PoseCNN-PyTorch [24] by NVIDIA Research Projects. We call our network “Bingham-PoseCNN” for convenience. Fig. 2 shows the overview of Bingham-PoseCNN. “Ours” in the figure is the point of change compared to the original PoseCNN. We changed just the dimension of the final FCN layer from 4 to $d$ ($d$ varies from 7 to 10).

On the one hand, our Bingham NLL loss function is described in Section IV. On the other hand, original PoseCNN used the following loss functions for training;

where $\mathcal{M}$ denotes the set of points on the mesh model of each object. They used PLoss if the object has no symmetry, and SLoss if the object has symmetry. While they annotated the symmetric property to each object, our method doesn’t need these annotations. In addition, our method doesn’t need mesh models of target objects.

We tested our model with the YCB-Video dataset which is the same dataset used in [1]. In this dataset, 80 videos for training, and 2949 keyframes for testing that are extracted from the rest 12 unused videos were provided. In addition, YCB-Video dataset contains 80000 synthetic images. We also used them for training.

We used ADD and ADD-S metrics to evaluate the performance of our pose estimator. Let $(R,\bm{t})$ be a pair of the groundtruth rotation $R$ and translation $\bm{t}$, and $(\widehat{R},\widehat{\bm{t}})$ be a pair of the estimated rotation and translation. Then, ADD and ADD-S are defined as below:

where $\mathcal{M}$ is the set of the $m$ sampled points from 3D mesh model’s surface. These metrics are the same as that used in PoseCNN [1].

Table II is evaluated only with the mode quaternion. Our method can extract information about the ambiguity or uncertainty of inferrence result. Inferred results are probability distribution so we may interpret them in a several way. We evaluate the result in the two interpretation: confidence and shape ambiguity.

Fig. 5 shows synthetic images of a red cup. It shows the groundtruth of the angles of rotation around $z$-axis, instead of the quaternions. The inferred distributions with their 10D parameters are also shown. We will use this figure for explaining the learning mechanism of our network.

Our model is mathematically represented as below:

where $I_{i}$ is an input image and $\bm{\theta}_{i}$ is a inferred parameter of distribution. $F$ is our network to be trained. Suppose that we have pairs $\{I_{i},\bm{q}_{i}\}_{i=1}^{N}$ whose images are similar to each other but whose groundtruth quaternions are all different. In Fig. 5, the middle column is corresponding to this circumstance. In this situation, there is a $\bm{\theta}$ satisfying

Let $M$ be a function that transforms a given parameter vector to a parameter matrix of Bingham distribution. Then our problem becomes

Find $\bm{\theta}$ that minimize $\displaystyle\sum_{i=1}^{N}\mathcal{L}(M(\bm{\theta}),\bm{q}_{i})$.

By solving this problem, we finally get $\bm{\theta}$ which is optimized to the given all quaternions $\bm{q}$. The problem is equivalent to “solve the maximum likelihood estimation (MLE) problem for each $M(f(I_{i}))$, given $\bm{q}_{i}$”. This means that the resulting parameter has information about the distribution of quaternions that share the similar views.

In Fig. 5, the inferred results in the left and right column are both concentrated because quaternions that gives the similar view are close to each other. In contrast, the result in the middle column is zonally spread because the quaternions sharing the view is widely spread. Our network learns the parameter that covers rotations sharing the similar view.

In Table II, the objects with discrete symmetry, such as 036_wood_block and 052_extra_large_clamp, got relatively low score in ADD and ADD-S. This is because a single Bingham distribution cannot capture the ambiguity with multiple modes well. We can improve score by introducing mixture Bingham representation which is introduced in such as [7]. Our NLL loss function is easy to extend to them, compared to non-NLL losses such as the QCQP loss presented in [6].