Reflection and Rotation Symmetry Detection via Equivariant Learning

Rotation and translation symmetries are often formulated as periodic signals and detected by autocorrelation [24, 28] in the spatial domain, spectral density [22, 23] and angular correlation [20] in the frequency domain. Meanwhile, there is a consistent need for an affine-invariant or equivariant feature descriptor in detecting symmetries, as matching the local descriptors is the most common solution. Loy and Eklundh [32] use SIFT [31] descriptors and normalize each descriptor by its dominant orientation. Cho and Lee [3] also use SIFT [31] to match feature pairs and detect symmetry by discovering the clusters of the nearby matches. Contour and edge features [39, 1, 37, 42, 43] are also useful for determining the boundary of the symmetric object. Lee and Liu [23] propose to solve affine-skewed rotation symmetry group detection by rectifying the skewed patterns. In this paper, we tackle the task by data-driven approach with our proposed dataset. Also, we use the dihedral group to interpret the symmetries as in many symmetry detection literatures [22, 23].

Recently proposed methods [11, 41, 13, 38] predict pixel-wise symmetry scores. Fukushima and Kikuchi [11] build a neural network to extract edges from images and detect reflection symmetry. Tsogkas et al. [41] construct a bag of features using histogram, color, and texture for each pixel and adopt multiple instance learning when training the model. Gnutti et al. [14] take two stages of computing the symmetry score for each pixel using patch-wise correlation and validating the obtained candidate axes using gradient direction and magnitude. Funk and Liu [13] are the first to adopt deep CNNs for detecting reflection and rotation symmetries. Seo et al. [38] propose a polar self-similarity descriptor for better rotation and reflection invariance. A specially designed Polar Matching Convolution (PMC) performs region-wise feature matching to compute the symmetry score, but the model relies heavily on the CNNs. To discover consistent symmetry patterns w.r.t. geometric transformations of rotation and reflection, we deploy group equivariant neural networks in our symmetry detection model.

A group $(G,\cdot)$ is a set $G$ with a binary operation $\cdot$, where its elements are closed under the operation. A group has unique identity element and inverse element and also satisfies the associativity. Equivariance of a map $f:X\xrightarrow{}Y$ is formalized using a group $G$ and two $G$-sets $X$ and $Y$, where $G$-set is a mathematical object consisting of a set $S$ and a group action of $G$ on $S$. A map $f$ is said to be equivariant iff

$\displaystyle f(g\cdot x)=g\cdot f(x),$

(1)

for all $x\in X$ and all $g\in G$. In 2D image domain, we focus on an euclidean group E(2), which is a group of plane $\mathbb{R}^{2}$ isometries of translations, rotations, and reflections.

The affine transformations of the 2D or 3D coordinates are easily done by matrix multiplications. Unlike low-dimensional feature vectors, the reflection or rotation transformation of the high-dimensional feature vectors are non-trivial. The first step to build a steerable convolution is to define a steerable feature field $f:\mathbb{R}^{2}\xrightarrow{}\mathbb{R}^{c}$ that maps a feature vector $f(x)\in\mathbb{R}^{c}$ with each point x of a base plane. Given a group $G$, a group representation $\rho:G\xrightarrow{}\mathrm{GL}(\mathbb{R}^{c})$ specifies the transformation law for shuffling the $c$ channels of each feature vector, where a general linear group $\mathrm{GL}$ is the set of $c\times c$ invertible matrices. Thus, applying transformation on a feature map not only moves the target vectors to the new positions but also shuffles each vector via $\rho(g)$ where $g\in G$. The group representations of E(2) group are presented in  [44].

To preserve the transformation law of the steerable feature spaces in CNNs, equivariance under the group actions is required for each network layer. Convolutions with the restricted G-steerable kernels [44] provide an equivariant linear mapping between the steerable feature spaces. The input and output of the G-steerable layers are the feature fields with their group representations $\rho_{\mathrm{in}}(g)\in\mathbb{R}^{c_{\mathrm{in}}\times c_{\mathrm{in}}}$ and $\rho_{\mathrm{out}}(g)\in\mathbb{R}^{c_{\mathrm{out}}\times c_{\mathrm{out}}}$, where a group element $g$ is specified. A kernel $k:\mathbb{R}^{2}\xrightarrow{}\mathbb{R}^{c_{\mathrm{out}}\times c_{\mathrm{in}}}$ that transforms under $\rho_{\mathrm{in}}$ and $\rho_{\mathrm{out}}$ becomes G-steerable when satisfying a kernel constraint of

$\displaystyle k(gx)$

$\displaystyle=\rho_{\mathrm{out}}(g)k(x)\rho_{\mathrm{in}}(g^{-1}),$

(2)

for every $g\in G$ given $x\in\mathbb{R}^{2}$. E(2)-equivariant CNNs solve this constraint to get a basis of the steerable kernels and comput the convolutional weights, which results in the smaller learnable parameters compared to the CNNs.

Since we aim to establish both reflection and rotation symmetry, we employ an E(2)-equivariant CNNs of dihedral group $\mathrm{D}_{N}$, which contains $N$ discrete rotations by angles multiples of $\frac{2\pi}{N}$ and reflections. The encoder $\mathrm{Enc}$ consists of an E(2)-equivariant [16, 44] ResNet [18] and an Atrous Spatial Pyramid Pooling(ASPP) [2] module. The decoder $\mathrm{Dec}$ is a 3-layer convolution module. The encoder and decoder designs follow [38] except that all convolution layers are substituted by the E(2)-equivariant convolution layers. During the forward computation, the feature fields are transformed into the predefined fields of the group $\mathrm{D}_{N}$. For the predictions ${\mathbf{S}}^{\mathrm{ref}}$ and ${\mathbf{S}}^{\mathrm{rot}}$, the feature fields of the decoders $\mathrm{Dec}^{\mathrm{ref}}$ and $\mathrm{Dec}^{\mathrm{rot}}$ are transformed back to the scalar fields.

Instead of a direct regression of the symmetry score maps [13, 38], we perform relevant subtasks that can lead to the final prediction. The proposed auxiliary tasks are the pixel-wise classification of the orientation (angle) of the reflection axis and the number of rotation folds. For simplicity, we assign the orientation into $N^{\mathrm{ref}}$ bins dividing 180 degrees. The ground-truth orientation $\mathbf{S}^{\mathrm{ref}}_{\mathrm{gt}}$ is then quantized to be one-hot. The auxilary rotation label $\mathbf{S}^{\mathrm{rot}}_{\mathrm{gt}}$ is the order of the rotational symmetry, which is annotated with a positive integer in the case of the discrete rotational symmetry. We allocate ’0’ to the continuous group since its ground-truth order is infinite. The size of the unique set of the rotation orders in the dataset is denoted as $N^{\mathrm{rot}}$. Meanwhile, we add background classes for the pixels that are neither the axis nor the center. Therefore, the classifiers predict the scores of channel of $N^{\mathrm{ref}}+1$ and $N^{\mathrm{rot}}+1$. The classification logit ${\mathbf{S}}\in\mathbb{R}^{H\times W\times(N+1)}$ is obtained by

$\displaystyle{\mathbf{S}}$

$\displaystyle=\mathrm{Dec}(\mathrm{Enc}(\mathbf{I})).$

(3)

The encoder $\mathrm{Enc}$ is shared while the decoders $\mathrm{Dec}^{\mathrm{ref}}$ and $\mathrm{Dec}^{\mathrm{rot}}$ are task-specific. Note that we set the group orientation $N$ as the same as $N^{\mathrm{ref}}$ to further exploit the equivariance of the equivariant networks.

The predicted score maps of the symmetry axes of the corresponding orientation are present in the $N^{\mathrm{ref}}$ foreground channels of the estimated orientation ${\mathbf{S}}^{\mathrm{ref}}$, whereas the background channel contains the background pixels. Similar to reflection, the $N^{\mathrm{rot}}$ foreground channels are the score maps of the rotation centers with that number of folds. We aggregate the sum of the foreground logits $\mathbf{P}\in\mathbb{R}^{H\times W\times 1}$ and the intermediate prediction ${\mathbf{S}}\in\mathbb{R}^{H\times W\times(N+1)}$ to compute the final prediction ${\mathbf{Y}}\in\mathbb{R}^{H\times W}$ as

	$\displaystyle{\mathbf{P}}_{h,w}$	$\displaystyle=\sum^{N}_{k=1}\frac{\exp{({\mathbf{S}}_{h,w,k})}}{\sum_{c}\exp{({\mathbf{S}}_{h,w,c})}},$		(4)
	$\displaystyle{\mathbf{Y}}$	$\displaystyle=\mathrm{conv}_{\mathrm{G}}([\mathbf{P}||\mathbf{S}]).$		(5)

Note that $||$ denotes the concatenation operation along the final channel dimension.

The early reflection symmetry datasets [4, 27] contain small number of images with few reflection axis and rotation center. Recently proposed BPS [13] and LDRS [38] are large enough to train deep architectures, but the reflection axes still lack of diversity in terms of the length and orientation. For example, objects with multiple symmetry axes are often annotated only by a single dominant axis. Furthermore, no existing reflection symmetry datasets take into account the continuous symmetry group, such as a circle with an infinite number of reflection symmetry axes. For the rotation symmetry, the annotations of BPS [13] are limited to the rotation centers while the pioneer unsupervised methods [22, 23] tackle the rotation folds also.

To address the concerns mentioned above, we present a new dataset for reflection and rotation symmetry detection that includes a wide range of geometries. We integrate 239 images of NYU [4] and 181 images of SDRW [27], and collect 2,080 images from COCO [26] dataset. Both reflection and rotation annotations are labeled for each image, and we remove images without any labels. As a result, DENDI contains 2,493 and 2,079 images for reflection and rotation split, respectively. The sizes of the symmetry detection datasets are compared in  Tab. 1. To add reflection axes with diverse length and orientation, we annotate objects in common shapes such as circle, ellipse, and polygons, as well as the part-level symmetries. Also, the annotators are encouraged to exhaustively label the symmetries for each object, including the non-dominant ones, e.g. the diagonals of a square. For the reflection symmetry of a continuous symmetry group, we annotate with ellipse-shaped masks to represent an infinite number of line axes. For the rotation symmetry, we additionally collect the number of rotation folds for each rotation center. As a result, the annotations in DENDI are denser and more diverse compared to the ones in the existing datasets.

A reflection symmetry axis is defined as a line formed by two points following [12, 13, 38, 4, 27]. In contrast to the existing datasets, we now account for circular objects. A circular object, which is equivalent to a filled circle, has an infinite number of reflection symmetry axes through its center. We propose to annotate the circular objects with 5 connected points, resembling the shape of the Arabic number ’4’. We draw a ’4’-shaped annotation from the circle’s center towards the circle’s boundary in the up, down, left, and right directions. The annotation rules and visualizations of the reflection symmetry for generic shapes are visualized in  Fig. 3 (a) and  Fig. 4 (a), respectively.

We collect the rotation center coordinate, the object’s boundary, and the number of folds (N) for each object. A circular object is in a continuous rotation group with infinite folds. Thus, we set the N as 0 for simplicity. We categorize the objects into ellipses and polygons based on the their shape. Circular or elliptical objects are marked with ’4’-shaped annotation. We annotate the polygons of V vertices with (V+1) consecutive points. From the center of the object, we take the vertex closest to 12 o’clock as the 2nd point and link the vertices of a convex polygon counter clockwise. Note that the number of the vertices (V) and the number of the folds (N) not always match. The annotation rules and visualizations of the rotation symmetry for generic shapes are visualized in  Fig. 3 (b) and  Fig. 4 (b), respectively.

Histograms for scale and orientation of the reflection axes are presented in  Fig. 5 (a) and (b). For scale, we measure the length of each line and normalize it with the length of the image diagonal. With two points on the line annotation, we can also compute its orientation (tangents). The y-axis of  Fig. 5 (a) and (b) is the ratio of the number of the axes over the total number of the axes. The number of axes increases when the length of each axis decreases, reflecting the characteristics of the real-world environment. In addition, it can be interpreted that part-level symmetry is densely annotated compared to other datasets. Note that our dataset ranks first at all orientations except for the three orientations that are closest to the vertical direction. Despite the fact that LDRS [38] is also based on COCO [26], the distribution of the orientation of the axis is more diverse in our case due to the fact that we particularly request the annotators not to omit the non-dominant axes.

Histograms of the rotation fold and the number of the rotation centers are illustrated in  Fig. 5 (c) and (d). The three most common folds in the dataset are 2, 0 (continuous), and 4. The result is predictable as there are many rectangles, circles, and squares in the image. Note that the dataset contains a notable number of objects of fold 8, which are mostly the ’STOP’ signs in the road. The complexity of our rotation symmetry dataset is high, as shown in  Fig. 5 (d). Even if we clip a few exceptions, a lot of rotation centers are marked in the images.

We use SDRW [27], LDRS [38], and DENDI to evaluate the reflection symmetry detection model. We follow the training and evaluation settings of PMCNet [38] in  Tab. a.1 for additional use of NYU [4] and the synthesized images. For rotation, we only use DENDI which contains the images of SDRW [27] rotation dataset.

To evaluate EquiSym, we use F1-score computed with the precision and recall as $\frac{2\times\mathrm{prec}\times\mathrm{rec}}{\mathrm{prec}+\mathrm{rec}}$. A convention [13, 38, 41] for measuring the score of the output score map is morphological thinning [34] and an off-the-shelf pixel-matching algorithm to compare with the ground-truth lines, which are also pixel-width. In contrast to the existing datasets, we take the circular objects into account, which are annotated with filled circles. As DENDI contains annotations of filled circles, the thinning operation shrinks to a single-pixel dot. Therefore, it is inevitable to come up with a new way to evaluate the predicted score maps. We dilate the ground-truth score maps of reflection and rotation symmetries with a maximum distance of 5 pixels following [13] where they enlarge the ground-truth dots to 5-pixel radius circles around the rotation centers. The predicted score map is also dilated to make the result of the ground-truth itself become 1. The true-positives are then computed by pixel-wise comparisons.

As a backbone network, we adopt the ReResNet implementation of ReDet [16] of depth 50, which is based on PyTorch [36] and e2cnn [44]. The number of the layers and the structure are the same as the vanilla ResNet [18]. We pretrain the ReResNet50 for the image classification task in imagenet-1k [10] following the procedures in  [16]. For the symmetry group to initialize the equivariant networks, we use a dihedral group of eight orientations ($\mathrm{D}_{8}$). To provide multi-scale contexts, we also deploy the Atrous Spatial Pyramid Pooling module (ASPP) [2] which we re-implemented the module by replacing all the vanilla convolutions to E(2)-equivariant convolutions [44]. The number of the classes $N^{\mathrm{ref}}$ is 8 and $N^{\mathrm{rot}}$ is 21. We train EquiSym for 100 epochs with an initial learning rate of 0.001 using the Adam [21] optimizer with a batch size of 32. For details, refer to the supplementary material.

We study the effectiveness of the group-equivariant convolution in  Tab. 2. With the help of the equivariant convolutions the F1 scores of 55.1 and 17.7 increase to 63.1 and 21.2 for EquiSym-$\mathrm{ref}$ and EquiSym-$\mathrm{rot}$, respectively.

To enhance the intermediate representation, we perform a relevant subtask for each branch of symmetry detection and compare them in  Tab. 2. Without extra labels, the reflection-only model achieves the F1 score of 64.5, which is greater than the 63.1 obtained by training solely with the final task. The orientation estimation also requires rotation equivariance, which enhances the intermediate features. Rotation symmetry detection, on the other hand, requires extra annotation for the auxiliary task as the original labels are a set of dots. The rotation invariant subtask of classification of the rotation folds (N) compresses the information of the intermediate feature so that it can increase the F1 score from 21.2 to 22.5.

We investigate the effect of a joint training of the reflection and rotation symmetries in  Tab. 2. When training EquiSym-$\mathrm{joint}$, the loss $\mathcal{L}$ is computed for both reflection and rotation symmetries. Joint symmetry detection network that is trained only with the final task achieves comparable F1 scores of 62.2 and 22.1 for reflection and rotation symmetry, respectively. However, the auxiliary task do not increase the accuracy of the reflection symmetry in joint training scenario. One probable explanation is that only orientation estimation of the reflection symmetry axis requires rotation equivariance while the fold classification only necessitates rotation invariance, resulting in network imbalance.