A Note on Graph-Based Nearest Neighbor Search

Nearest neighbor search among database vectors for a query is a key building block to solve problems such as large-scale image search and information retrieval, recommendation, entity resolution, and sequence matching. As database size and vector dimensionality increase, exact nearest neighbor search becomes expensive and often is considered impractical due to the long search latency. To reduce the search cost, approximate nearest neighbor (ANN) search is used, which provides a better tradeoff among accuracy, latency, and memory overhead.

Roughly speaking, the existing ANN methods can be classified into space-partitioning algorithms and graph-based ones¹¹1Please note the categorization is not fixed/unique. The space-partitioning methods further fall into three categories - the tree-based, production quantization (PQ) and locality sensitive hashing (LSH)[1, 2]. Recent empirical study shows that graph-based ANN search algorithms are more efficient than the space-partitioning methods such as PQ and LSH, and thus have been adopted by many commercial applications in Facebook, Microsoft, Taobao and etc. [3, 4, 5].

While a lot of empirical studies validate the efficiency of graph-based ANN search algorithms, not much attention has been paid to a more fundamental question: why graph-based ANN search algorithms are so efficient? And which data property affect the efficiency and how? Two recent papers analyze the asymptotic performance of graph-based methods for datasets uniformly distributed on a $d$-dimensional Euclidean sphere [6, 7]. The worst-case analysis shows that the asymptotic behavior of a greedy graph-based search only matches the optimal hash-based algorithm [8], which is far worse than the practical performance of graph-based algorithms and thus could not answer these questions.

A few conceptual graph models such as Monotonic Search Network Model [9], Delaunay Graph Model [10, 11] and Navigable Small World Model [12, 13] are proposed to inspire the construction of ANN search graphs. As will be discussed in Section II, none of them can explain the success of graph-based algorithms either. Actually, the vast majority (if not all) of practical ANN search graphs uses approximate $K$NN graph as the index structure instead of the conceptual models due to time or space constraints, and thus is fully devoid of the theoretical features provided by these models.

In this paper, we argue that, for a specific dataset, the clustering coefficient [14] of its $K$NN graph is an important indicator on how efficiently graph-based algorithms work. The clustering coefficient of $K$NN graph defines the probability of neighbors of a point are also neighbors. Comprehensive experimental results reveal that higher the clustering coefficient is, more efficiently the graph-based algorithms will perform. Since clustering coefficient is data dependent, graph-based algorithms perform rather worse for datasets such as Random with very small clustering coefficient, whereas do well in datasets such as Sift and Audio with greater ones.

We also study how clustering coefficient affects the performance. The analysis of the complex network indicates that large clustering coefficient leads to high global connectivity [15]. Our insight is that, instead of the global connectivity, the local graph structure is more crucial for high ANN search recall. Particularly, we observed that, for datasets with large clustering coefficient, most of the $k$NN of some given query²²2This query can be in or not in the dataset lie in the maximum strongly connected component (SCC) in the subgraph composed of these $k$NN. Moreover, we show that the search procedure actually consists of two phases, the one outside the maximum SCC and the one in it, in contrast to the common wisdom of single or multiple path search models. Then, we proved that the commonly used graph search algorithm is guaranteed to visit all $k$NN in the maximum SCC under a mild condition, which suggests that the size of the maximum SCC determines the answer quality of $k$NN search. This sheds light on the strong positive correlation between the clustering coefficient and the result quality, and thus answers the two aforementioned questions.

To sum up, the main contributions of this paper are:

• •

  We introduce a new quantitative measure Clustering Coefficient of $K$NN graph for the difficulty of graph-based nearest neighbor search. To the best of our knowledge, this is the first measure which could explain the efficiency of this class of important ANN search algorithms.

• •

  The conceptual models such as MSNETs and Delaunay graphs claim that NN could be found by walking in a single path. Instead, we found that the search procedure is actually composed of two phases. In the second phase the algorithm traverse the maximum SCC of $k$NN for a query, of which the size is a determining factor for answer quality, i.e., recall.

• •

  We proved that the graph-based search algorithm is guaranteed to visit all points in the maximum SCC once entering it. Extensive empirical results over a comprehensive collection of datasets validate our observations.

We believe that this note could provide a different perspective on graph-based ANN search methods and might inspire more interesting work along this line.

In graph theory, the clustering coefficient is a measure of the degree to which nodes in a graph tend to cluster together, which has been used successfully in its own right across a wide range of applications in complex networks. To name a few, Burt uses local clustering as a probe for the existence of so-called “structural holes” in a network, and Dorogovtsev et al. found that $C_{i}$ falls off with $k_{i}$ approximately as $k_{i}^{-1}$ for certain models of scale-free networks [15], where $k_{i}$ is the degree of $v_{i}$.

There are different ways to define clustering coefficient. In this paper, we adopt the commonly used definition given by Watts and Strogatz [14]. The local clustering coefficient $C_{i}$ of a vertex $v_{i}$ is defined as

To calculate $C_{i}$, we go through all distinct pairs of vertices that are neighbors of $v_{i}$ in the network, count the number of such pairs that are connected to each other, and divide by the total number of pairs $\frac{k_{i}(k_{i}-1)}{2}$. Figure 3 illustrates the definition of the local clustering coefficient $C_{i}$. The degree of $v_{i}$ is 4 and there exists two edges between its neighbors. Hence, by definition $C_{i}=2\div\frac{4\times(4-1)}{2}=\frac{1}{3}$.

The clustering coefficient for the whole network is the average

The local clustering coefficient $C_{i}$ of a vertex $v_{i}$ describes the likelihood that the neighbours of $v_{i}$ are also connected, i.e., the probability that two randomly selected neighbors of $v_{i}$ are neighbors with each other. Roughly speaking, it tells how well the neighborhood of the node is connected. If the neighborhood is fully connected, the local clustering coefficient is 1 and a value close to 0 means that there are hardly any connections in the neighborhood. If most of the nodes in the network have high clustering coefficient, then the network will probably have many edges that connect nodes to each other.

Clustering coefficient of $K$NN graphs depends on $K$ and the intrinsic feature of datasets. Table I lists the clustering coefficients for various $K$ under three typical datasets As one can see, the larger $K$ is, the greater clustering coefficient will be. Moreover, the relative order of the clustering coefficient is stable independent of $K$. In the sequel, we will use the clustering coefficient in the case of $K=50$ as the default since $K$ cannot be too large due to the index space constraint.

Our key observation is that the clustering coefficient of $K$NN graphs is a informative measure for the efficiency of the graph-based ANN search methods. In this paper, a $K$NN graph is defined as a graph such that for each vertices $v$, there exists bi-directional edges with its $K$ most nearest neighbors. This model is reasonable because practical graph-based algorithms such as HNSW and NSG always add bi-directional links between a point and its $K$NN as much as possible [22, 5]. Table II lists the statistics of the datasets, the clustering coefficients of $K$NN graph in increasing order, the recalls of the top-$50$ query and the average number of hops in the graph for a collection of datasets under HNSW and NSG, the two state-of-the-art graph-based algorithms³³3statistics are collected over 1000 random queries. For both methods, the maximum out-degrees (MOD) is 70 and the parameter $L$, which control the number of hops in the graph, is set to 50. Interesting observations can be made as follows:

• •

  NSG consistently outperforms HNSW in recall with slightly greater average number of hops, which approximately translates to the number of distance evaluation since the MODs are identical for both algorithms. This observation agrees with the results reported in [5].

• •

  A more interesting observation is that, with around the same number of average hops in the graph, clustering coefficient and recall are strongly correlated. Particularly, The Pearson correlation coefficients between the clustering coefficient and recall for NSG and HNSW are 0.794 and 0.762, respectively. Besides, independent of the data cardinality and dimensionality, high clustering coefficient (greater than 0.12) often leads to high recall whereas low clustering coefficient (lower than 1.0) results in low recall. As an extreme example, the clustering coefficient of Random dataset is only 0.00074 and thus makes graph-based algorithms become very inefficient. One reason that the recall of NSG is greater than that of HNSW is that the quality of neighbors of NSG is better than HNSW, that is, NSG is much closer to an exact $K$NN graph than HNSW. Please note that the datasets are comprehensive enough in terms of size, dimensionality and data types (images, text, audio and synthetic). Detailed description of these datasets can be found in [24]⁴⁴4https://github.com/DBWangGroupUNSW/.

These observations suggest that the clustering coefficient are a promising measure for the efficiency of graph-based algorithms. Intuitively, higher the clustering coefficient of $K$NN graph is⁵⁵5$K$ should be as small as possible to reduce the memory footprint and query efficiency, the better the graph is connected, which means that graph connectivity has significant impact on the result quality of ANN search. To have an in-depth understanding of how connectivity affects the search performance, we scrutinized the graph traversal steps of a sample of queries and found that the local connectivity, instead of the global one, is the determining factor. To formally characterize the local connectivity, we propose the notion of maximum strongly connected neighborhood as follows.

Please note that the only requirement is the $k$ nearest neighbors of $v$ belongs to $V$ and $v$ may be some point in or not in $V$. This definition makes our analysis supports out-of-sample queries.

Please note that $K$ and $k$ owns totally different meaning - $K$ is the link number per node of $K$NN graph and is determined in graph construction and $k$ is the search parameter and thus may vary according to the users’ requirement.

Figure 4 illustrates these definitions using a simple example. $o_{1}$ to $o_{5}$ is the top-5 NN of query $q$ and there are three undirected edges (equivalent to six directed edges) in $q$’s 5-neighborhood subgraph $\mathcal{G}_{5}(q)$. Three SCCs exists in $\mathcal{G}_{5}(q)$ and the maximum SCC $\mathcal{C}_{5}(q)$ is composed of point $o_{1}$, $o_{2}$ and $o_{3}$.

To show the impact of $\mathcal{C}_{k}(v)$ on the algorithm performance. Table III lists the 3 SCCs of largest sizes for 100 random $k$NN queries in the case of $k=50$ for three typical datasets. As we can see, the ratios of the size of $\mathcal{C}_{k}(v)$ (SCC1) to $k$ are very close the recall listed in Table II for three datasets, respectively. Other state-of-the-art algorithms, such as HNSW, exhibits similar trends.

To eliminate the bias caused by specific graph construction algorithms, we studied the exact $K$NN graph and found similar results. Table IV lists, for 100 random top-50 NN queries, the average sizes of the top-3 SCCs over Sift, Glove and Random. In this experiment, we only put a directed link from a point to its $K$NN and no link is added manually in the reverse direction, i.e., the $K$NN graph is directed. $K$ is set to 50. From Table IV we can see that, independent of specific graph-based algorithm, clustering coefficient also has a significant impact on the size of $\mathcal{C}_{k}(v)$.

We also examined undirected $K$NN graph, where bi-directional link is added manually between a point and its $K$NN. The trend listed in Table V is very similar to that of Table IV except that the sizes of $\mathcal{C}_{k}(v)$ are larger. This is because more links are added in the graph. Actually, practical graph-based algorithms lie somewhere between the undirected and directed $K$NN graph since they always try to add bi-directional links as long as the memory budget is enough. Please note exact $K$NN graphs are not practical because the unaffordable construction time and unlimimted maximum out-degree, which translates to too much memory cost.

In a nutshell, all these experiments demonstrate that clustering coefficient of $K$NN graph is an informative measure for the size of the maximum strongly connected neighborhood and the performance of graph-based algorithms over a specific dataset. Next, we will analyze how $\mathcal{C}_{k}(q)$ affects the recall for a given query $q$. Particularly, we will show that Algorithm 2, the striking algorithmic component for graph search, can effectively reach $\mathcal{C}_{k}(q)$ and identify all $k$NN $\in\mathcal{C}_{k}(q)$. This explains why greater clustering coefficient and larger size of $\mathcal{C}_{k}(q)$ lead to better performance.

The common wisdom about Algorithm 2 is as follows. Starting from the entry vertex $s$, which is chosen by random or using some auxiliary method, Algorithm 2 finds a directed path from $s$ to the query $q$, hoping that NN of $q$ are identified through the walk. Since only local information, i.e., adjacent vertices of the visited vertices, is used, this class of algorithms are termed as the decentralized algorithm [12]. Particularly, for ANN search, Algorithm 2 first follows the out-edges of $s$ to get its immediate neighbors, and then examines the distances from these neighbors to $q$. The one with the minimum distance to $q$ is selected as the next base vertex for iteration. The same procedure is repeated at each step of the traversal until Algorithm 2 reaches a local optima, namely, the immediate neighbors of the base vertex does not contain a vertex that is closer to $q$ than the base vertex itself. Backtracking is used to jump out of the local optima and increase the odd to find the true NN. Recall that we name this search paradigm as multiple path search model.

Different from the traditional point of view, we observe that Algorithm 2 actually is composed of two phases. In the first phase, the algorithm starts with an initial point, walks the graph and encounters a point within $\mathcal{C}_{k}(q)$. In the second phase, the algorithm traverse $\mathcal{C}_{k}(q)$ and a small number of points not in $\mathcal{C}_{k}(q)$. Figure 5 depicted the two phase search procedure. Theorem 1 proves that Algorithm 2 is guaranteed to find all points in $\mathcal{C}_{k}(q)$ under a mild condition.

Theorem 1 suggests a different perspective in understanding the graph-based methods. Rather than searching a single path (without backtracing) or multiple paths (with backtracking) in the graph, the search algorithm actually traverses a strongly connected neighborhood around the query. In other words, high quality $\mathcal{C}_{k}(q)$, together with Algorithm 2, offers the salient performance. The analysis in Section III reveals the quality of $\mathcal{C}_{k}(q)$ are data dependent and closely related to the clustering coefficient of $K$NN graphs. Therefore, there exists significant performance disparity for different datasets and we could use the clustering coefficient of $K$NN graph as a meaningful measure for the efficiency of the graph-based methods.

It is possible that there exists a few vertices adjacent with vertices in $\mathcal{C}_{k}(q)$, which is not in $\mathcal{C}_{k}(q)$ but closer to $q$ than some vertices in $\mathcal{C}_{k}(q)$. In this case, the algorithm may also visit such vertices, and the answer quality will be higher than just traversing $\mathcal{C}_{k}(q)$ since more closer vertices outside $\mathcal{C}_{k}(q)$ are visited.

The probability of the search algorithm getting into $\mathcal{C}_{k}(q)$ is exponentially increasing with $L^{\prime}$, the number of being trapped into a local optima and getting back to a distant point before entering $\mathcal{C}_{k}(q)$, which is expressed as follows. $p_{i}$ is the probability of getting into $\mathcal{C}_{k}(q)$ along a single path.

The rigorous calculation of $P$ is infeasible. Empirically, for datasets with relatively large clustering coefficients we observed that (1) Algorithm 2 can quickly reach $\mathcal{C}_{k}(q)$, and (2) the path length of the first phase is far shorter than that of the second phase. Table VI, Table VII and Table VIII list the numbers of hops in Phase 1 and Phase 2, the size of $\mathcal{C}_{k}(q)$, the fraction of points in $\mathcal{C}_{k}(q)$ that are visited in Phase 2 and the number of true top-$k$ NN not in $\mathcal{C}_{k}(q)$ that are found during Phase 2 (denoted by $|\overline{\mathcal{C}_{k}(q)}|$) for Sift, Glove and Random, respectively. NSG is used and the statistics of five random query are reported. Please note that HNSW and exact $K$NN graph exhibit the similar trends thus we do not report results for them. Several interesting observations are made:

• •

  As we can see, independent of datasets, the two-phase search model is applicable to all queries listed. As proved in Theorem 1, once the search algorithm enters Phase 2, all points in $\mathcal{C}_{k}(q)$ will be visited, which demonstrates the importance of the quality of $\mathcal{C}_{k}(q)$.

• •

  Besides the true top-$k$ NN in $\mathcal{C}_{k}(q)$, other true top-$k$ NN also probably be visited during Phase 2, especially for Glove where the size of $\mathcal{C}_{k}(q)$ is relatively small. This is mainly caused by the search algorithm jumps into a smaller SCC or visits $k$NN that only own a single directed edge with the maximum SCC.

• •

  For Sift and Glove, where the size of $\mathcal{C}_{k}(q)$ far greater than that of Random, the second phase dominates the search cost and the algorithm jumps into $\mathcal{C}_{k}(q)$ in a very small number of steps. In contrast, it is very hard for the algorithm to find a true top-$k$ NN for Random since the size of $\mathcal{C}_{k}(q)$ is too small (in most case is equal to 1). For example, $q_{2}$ and $q_{5}$ do not enter Phase 2 and didn’t find any true NN. As a result, the recall of Random is very low.

To train reader’s intuition, Figure 6 illustrates the search procedure of a top-10 query for Sift dataset with NSG. Green point is the query and red points denote the true top-NN in the maximum SCC, which are strongly connected. Dashed lines in blue with single or double arrows represents the the directed or bi-directional edges between points. The solid arrowed lines in yellow depict the search path during $k$NN search. As we can see, starting with the entry point, the algorithm jumps into the maximum SCC in three steps. After traversing the maximum SCC which consists of six true NNs, it continues the search by visiting one true NN (in black) and four other points before the termination condition is met. Since $k$ is small in this example, the size of the maximum SCC is not that large. This can be informally explained by the fact that the connectivity increases as the number vertices become large under the same edge connection probability using random network theory [15].

The case of small $k$: Users may be only interested in a small number of nearest neighbors of $q$, say $k$ ranging from 1 to 10. In this case, the size and quality of $\mathcal{C}_{k}(q)$ is not that good to achieve high recall. To get precise results, $L$ is often set greater than $k$, say 50-200. The net effect is that the search algorithm actually visit $\mathcal{C}_{L}(q)$, which consists the most points of top-$L$ NN if the clustering coefficient is large enough, and then Algorithm 2 identify the best $k$ results and output them.

This paper takes the first step to shed light on why the graph-based search algorithms work so well in practice and suggests that the clustering coefficient of $K$NN graph is an important measure for the efficiency of these algorithms. Detailed analysis is also conducted to show how clustering coefficient affects the local structure of $K$NN graphs. A few open problems still exists. For example, formal analysis under some simplified data model is important to have more rigorous understanding of the graph search procedure.

The work reported in this paper is partially supported by NSFC under grant number 61370205, NSF of Xinjiang Key Laboratory under grant number 2019D04024.