Deep Extrinsic Manifold Representation for Vision Tasks

An intuitive embedding choice for the matrix Lie group is to parameterize each matrix entry. For the special orthogonal group $O(n)$ of dimension $n$, the natural embedding can be given by $J_{O}:O(n)\rightarrow\mathbb{R}^{n^{2}}$ and $J_{SO}:SO(n)\rightarrow\mathbb{R}^{n^{2}}$. For its inverse, $J_{O}^{-1}$ which aims to produce $n$ orthogonal vectors, where Gram Schmidt orthogonalization (Zhou et al., 2019) and its variations are common ways to reparameterize orthogonal vectors. Singular Value Decomposition (SVD) is another convenient way to produce an orthogonal matrix (Levinson et al., 2020):

where $U$ and $V$ come from singular value decomposition $\mu=UDV^{\top}$ with elements arranged according to the descending order of singular value, and $H$ is a diagonal matrix $I_{n}$ with the last entry replaced by $-1$. Specifically for $SO(3)$, $J^{-1}_{SO(3)}$ could also be produced by cross product.

As for the special Euclidean group, the group of isometries of the Euclidean space, i.e., the rigid body transformations preserving Euclidean distance. A rigid body transformation can be given by a pair $(A,t)$ of affine transformation matrix $A$ and a translation $t$, or it can be written in a matrix form of size $n+1$. For instance, a special Euclidean group can be regarded as the semidirect product of a rotation group $SO(n)$ and a translation group $T(n)$, $SE(n)=SO(n)\ltimes T(n)$.

Specifically for orthogonal matrices on $SO(3)$, the invertible embedding $J$ can be more convenient via cross-product operation $\times$. For a 6-dimensional Euclidean vector $x=[x_{a},x_{b}]$ as network output, where $x$ is the concatenation of $x_{a}$ and $x_{b}$, let $x_{c}=x_{a}\times x_{b}$, then $J^{-1}(x)=[x_{a}^{T},x_{b}^{T},x_{c}^{T}]$.

During training, at each iteration stage, a randomly chosen point cloud from $2,290$ airplanes is transformed by randomly sampling rotations and translations in batches. The rotations are sampled according to the models, i.e., the models producing axis angles are fed with rotations sampled from axis angles, and so on.

To validate the ability of the model to preserve geometrical structures, geodesic distance is the opted metric at the testing stage. (Zhou et al., 2019; Levinson et al., 2020) uses minimal angular difference to evaluate the differences between rotations, which is not entirely compatible with Euclidean groups, since it calculates the translation part and rotation part separately. For two rotations $R_{1},R_{2}$ and $R^{\prime}=R_{1}R_{2}^{-1}$ with its trace to be $tr(R^{\prime})$, there is $L_{angle}=\cos^{-1}((tr(R^{\prime})-1)/2)$. For intrinsic metric, the geodesic distance between two group elements are defined with Frobenius norm for matrices $d_{int}(M_{1},M_{2})=\|\log(M_{1}^{-1}M_{2})\|$, where $\log(M_{1}^{-1}M_{2})$ indicates the logarithm map on the Lie group. For extrinsic metric, we take Mean Squared Loss (MSE) between $J_{SE}(M_{1})$ and $J_{SE}(M_{2})$.

For models with Euler angle and axis-angle output, the predicted rotation and translation are organized in Euclidean form, and they are reorganized into Lie algebra $se(3)$. Finally, the distances are calculated in $SO(3)$ by exerting an exponential map on their $se(3)$ forms.

Similar as (Zhou et al., 2019; Levinson et al., 2020), the backbone for point cloud feature extraction is composed of a weight-sharing Siamese architecture containing two simplified PointNet Structures (Qi et al., 2017) $\Phi_{i}:\mathbb{R}^{N\times 3}\rightarrow\mathbb{R}^{1024}$ where $i=1,2$. After extracting the feature of each point with an MLP, both $\Phi_{1}$ and $\Phi_{2}$ utilize max pooling to produce a single vector $z_{1},z_{2}$ respectively, as representations of features across all points in $P_{r},P_{t}$. Concatenating $z_{1},z_{2}$ to be $z$, another MLP is used for mapping the concatenated feature to the final higher-order Euclidean representation $Y_{\mathbb{E}}$. Finally, Euler angle quaternion coefficients and axis-angle representations will be directly obtained from the last linear layer of the backbone network of dimension, $3,3,4$ respectively. For manifold output, the $SE(3)$ representation will be given by both cross-product and SVD, with details in Supplementary materials. The translation part is produced by another line of $\mathtt{fc}$ layer of 3 dimensions. Because MSE computation between two isometric matrices are based on each entry, total MSE loss is the sum of rotation loss and translation loss.

To validate the necessity of structured output space, firstly we compare the affine transformation estimating network with different output formations. The translation part naturally takes the form of a vector and the rotation part can be in the form of Euclidean representations and $SO(3)$ in matrix forms. The dataset is composed of generated point cloud pairs with random transformations, where the rotations are uniformly sampled from various representations, namely, Euler angle, axis-angle in $3$-dimensional vector space, and $SO(3)$. The translation part is randomly sampled from a standard normal distribution.

The advantage of manifold output produced by baseline DEMR is revealed in Table 1 and Figure 4, output via extrinsic embedding takes the lead across three statistics.

To evaluate DEMR over its improvement in generalization ability in comparison with unstructured output space, the training set only contains a small portion of the whole affine transformation space while the test set encompasses affine transformations sampled from the whole rotation space. To be fair in the sampling step across the compared representations, the sampling process is conducted uniformly in the representations spaces respectively. For models with axis-angle and Euler angle output, the input point cloud pair is constructed with rotation representation whose entries sampled $\mathrm{iid}$ from Euclidean ranges. For Euler representations, the lower bounds of the ranges are $\pi$, and the upper bounds are $\pi$ multiplied by $90\%,80\%,60\%,40\%,20\%$, while the test set is sampled from $[-\pi,\pi]$. The training set with axis-angle representations is also obtained by sampling Euclidean ranges. When constructing a training set on $SO(3)$, the uniform sampling process is conducted in the same way with axis-angle sampling, the segments are taken in ratios in the same way with the two aforementioned settings, and then exponential mapping is used to yield random samples on $SO(3)$. Facing unseen cases, DEMR has better extrapolation capability compared to plain deep learning settings with unstructured output space. As shown in Table LABEL:tab:_generalization, results from deep extrinsic learning show great competence despite the portion size of the training set.

A similar work sharing the same background comes from (Lohit & Turaga, 2017), which assumes the output of the last layer in the neural network to lie on the matrix manifolds or its tangent space. We take GrassmannNet in the former assumption in (Lohit & Turaga, 2017) as the baseline, where the training loss function is set to be Mean Squared Loss. For all the models to be compared, the ratio of the training set is $0.8$, and all of the illumination angles are adopted. In addition, the data preparation step also follows (Lohit & Turaga, 2017), and other training details are recorded in supplementary materials. On extended Yale Face Database B, each grayscale image of $168\times 192$ is firstly resized to $256\times 256$ as the input of CNN, while for subspace ground truth, they are resized to $28\times 28=784$ for convenience of computation.

The input face image $I_{i}^{j}$ indicates the $i_{th}$ face under the $j_{th}$ illumination condition, where $j=1,\ldots,64$ for Yale B dataset. If we adopt the top $d$ PCs, the output is assumed to be in the form of $\{E_{i}^{1},E_{i}^{2},\ldots,E_{I}^{d}\}$, $k=1\ldots,d$ and $<E_{i}^{k},E_{i}^{l}>=\delta_{kl}$ where $\delta_{kl}$ is a Kronecker delta function. Each PC is rearranged as a vector, thus $vec(E_{i}^{k})$ is of size $784\times 1$ and we define $U_{i}=[vec(E_{i}^{1}),vec(E_{i}^{2})\ldots vec(E_{i}^{d})]$, and the output of the network is $U_{i}U_{i}^{T}$. Most of the intrinsic distances defined on the Grassmannian manifold are based on principal angles (PA), for $P_{1},P_{2}\in\mathcal{G}(m,\mathbb{R}^{n})$, with SVD $P_{1}^{T}P_{2}=USV^{T}$ where $S=diag\{cos(\theta_{1}),cos(\theta_{2}),\ldots,cos(\theta_{m})\}$, PAs of Binet-Cauchy (BC) distance is $1-\Pi_{k=1}^{K}\cos^{2}\theta_{k}$ and PAs of Martin (MA) distance is $\log\Pi_{k=1}^{K}(\cos^{2}\theta_{k})^{-1}$. For comparison of effectiveness and training advantage, the deep intrinsic learning takes $D_{\mathcal{G}}(U_{gt},U_{out})=$ as both the training loss and testing metric, where $U_{gt}$ and $U_{out}$ indicate the ground truth and the network output respectively.

The results of illumination subspace estimation are given in Table 3. Measured by the average of geodesic losses, DEMR architecture achieves nearly the same performance as DIMR while drastically reducing training time till convergence. With batch size set to be 10, we record training loss after every 10 epochs and report approximately the epoch at which the training loss converged in the third row in Table 3. Besides, both DIMR and DEMR surpass GrassmannNet in the task of illumination subspace estimation, and DEMR still takes fewer epochs for training than GrassmannNet. In the first row of Table 3, on one of the test samples, the results from GrassmannNet, DIMR, and DEMR are displayed. In addition, the conformance of extrinsic and intrinsic metrics can be referred to as supplementary materials.