On Mitigating Hard Clusters for
Face Clustering

The standard way of construction the transition matrix is to compute the similarity between the deep features of pair-wise samples. The “similarity” can be the conventional cosine similarity or the learned similarity in more recent works such as [31, 14, 23]. To reduce the memory consumption of using GCNs [31, 23] for similarity learning, Pair-Cls [14] simply learns the similarity via pair-wise classification by deciding whether two points share the same identity. However, in Pair-Cls, all pairs are completely independent during training. We argue that the similarity between a point and one of its neighbors usually depends on the similarities between the point and its other neighbors. Therefore, in our work, we adopt the same pair-wise classification, i.e., using an MLP to predict the similarity, and besides that, we leverage a collaborative prediction manner by considering the similarities between each point (as an anchor) and its neighbors as a whole to improve the robustness of the prediction, similar to [19, 14].

Here, we elaborate a general formulation. For a sample point $i$, we first find its $K$-nearest neighbors denoted as $\text{nbr}_{i}=\{i_{1},\cdots,i_{K}\}$, and then generate the following token sequence:

Our similarity prediction model first takes $\tilde{x}_{i}$ as input, and outputs $K+1$ features after feature encoder $\mathcal{F}$:

where $\mathcal{F}$ can be Transformer [25], GCNs [31, 23] or a simple feature aggregation module [14] (aggregate features of neighbors and concatenate to the feature of anchor). Then, for each neighbor $i_{j}$, $j=1,...,K$, $t_{i}$ are concatenated with $t_{i_{j}}$ and fed into an MLP with Sigmoid function to estimate the probability of that $i$ and $i_{j}$ share the same identity:

Assuming $l_{ij}$ is the ground-truth label, $l_{ij}=1$ if $l_{i}=l_{i_{j}}$ and $l_{ij}=0$ vice versa. The total loss function is formulated as:

Finally, we assume $d_{i}=\sum_{j=1}^{N}\hat{s}_{ij}$ as the measure of the volume around point $i$, and generate the probability transition matrix $\bm{P}$ with each element as $p_{ij}={\hat{s}_{ij}}/{d_{i}}$. The size of $\bm{P}$ is $N\times N$. Please note that $\hat{S}$ is a sparse matrix where each row contains $K+1$ non-zero elements (itself and its top-$K$ nearest neighbors). Therefore, $\bm{P}$ is also sparse. We highlight that the above approach is not the only way to construct the transition matrix $\bm{P}$, and we show the results of using other approaches to obtain $\bm{P}$ in the experiment section.

In this section, we propose a new definition of the point-wise density, called NDDe, to alleviate the issue of small-size clusters. In the transition matrix $\bm{P}$, each element $\bm{P}_{ij}$ denotes the probability from one point $i$ to its specific neighbor $j$. It satisfies the conservation property, i.e., $\sum_{j}\bm{P}_{ij}=1$, which induces a Markov chain on $\bm{X}$. Denoting $\bm{L}=\bm{I}-\bm{P}$ as the normalized graph Laplacian, where $\bm{I}$ is the identity matrix. We can specify a diffusion process as follows,

where $\rho_{i}(t)$ is the density of point $i$ at $t$-th step. Starting from a uniformly initialized density, the diffusion process keeps distributing the density of each point to its $K$-nearest neighbors, following the corresponding transition probabilities in $\bm{P}$, until converged to a stationary distribution. The diffusion density thus can be induced as:

In this section, we introduces our new definition of the pair-wise distance, called TPDi, to solve the issue of varying sparsity in latent face clusters. TPDi depicts the similarity between two points based on their respective transition probabilities (in $\bm{P}$) to the common neighbors. Assuming $C_{ij}=\text{nbr}_{i}\cap\text{nbr}_{j}$ contains the common neighbors in the $K$-nearest neighbors of both point $i$ and $j$. TPDi between them is defined as:

We highlight that TPDi has three impressive properties: (1) By Cauchy-Schwarz inequality, we have $\left(\sum_{c\in C_{ij}}\sqrt{p_{ic}p_{jc}}\right)^{2}\leq(\sum_{c\in C_{ij}}p_{ic})(\sum_{c\in C_{ij}}p_{jc})\leq 1$, so it is easy to check $0\leq d_{ij}\leq 1$, which implies that $d_{ij}$ can be a valid metric. (2) $d_{ij}=0$ if and only if $p_{ic}=p_{jc}$ for all $c=1,...,N$, which implies that $d_{ij}$ is small when $i$ and $j$ share as many as common neighbors. It is consistent with the motivation of TPDi. (3) Compared with cosine distance, TPDi of negative pairs and positive pairs are better separated, regardless of cluster sparsity (Figure 5(c)(d)). So it is easier to choose a uniform threshold for TPDi.

The overall clustering procedure is summarized in Algorithm 1. In our implementation, we use an iterative method as an approximation of Eq. 3.

We compare the proposed method with a series of clustering baselines, including both traditional methods and learning-based methods. Traditional methods include $K$-means [18], HAC [24],DBSCAN [6], and ARO [20]. Learning-based methods include CDP [33], L-GCN [29], LTC [32], GCN (V+E) [31], Clusformer [19], Pair-Cls [14], STAR-FC [23], and Ada-NETS [28]. Since NDDe and TPDi can be incorporated into existing face clustering methods for better performance, we also incorporate them into GCN (V+E), Pair-Cls, and STAR-FC by using the three methods to obtain the transition matrix $\bm{P}$, which are denoted as GCN(V+E)++, Pair-Cls++, and STAR-FC++, respectively.

To demonstrate the effectiveness of NDDe and TPDi, we conduct an ablation study on MS1M 5.21M dataset, as shown in Table 5. All these four methods use the same transition matrix as described in Section 4.1. $M_{1}$ is our baseline model, which uses the standard density and cosine distance. $M_{2}$ is obtained by replacing the cosine distance in $M_{1}$ with TPDi, $M_{3}$ is obtained by replacing the standard density in $M_{1}$ with NDDe, and $M_{4}$ is the proposed method using both NDDe and TPDi as the density $\rho$ and distance $(d_{ij})_{N\times N}$ required by DPC. Table 5 shows that both NDDe and TPDi contribute to the final clustering performance. And the improvement brought by NDDe is more significant, which illustrates that NDDe is essential for the success of our method.

To further show the potential of our method in scaling up face recognition systems using large-scale unlabeled face images, we use our method to generate pseudo-labels for unlabeled face images and use them to train face recognition models. For a fair comparison, we adopt the same experimental setting as in [23, 32, 31]. We use a fixed number of labeled data and different ratios of unlabeled data with pseudo-labels to train face recognition models and test their performance on MegaFace benchmark [10] taking the rank-1 face identification accuracy with 1M distractors as metric. In Figure 8, the upper bound is trained by assuming all unlabeled data have ground-truth labels, and the other five curves illustrate that all the methods benefit from an increase of the unlabeled data with pseudo-labels. And it can be observed that our method consistently achieves the highest performance given any ratio of unlabeled data, and improves the performance of the face recognition model from $58.20\%$ to $80.80\%$, which is the closest to the upper bound.