Revisiting $k$-Nearest Neighbor Graph Construction on High-Dimensional Data : Experiments and Analyses

Let $D\subset\mathbb{R}^{d}$ be a data set that consists of $n$ $d$-dimensional real vectors. The KNNG on $D$ is defined as follows.

The KNNG of a data set with 6 data points is shown in Figure 2. For each node, it has two out-going edges pointing to its 2NNs. For simplicity, we refer to approximate KNNG as KNNG and approximate KNNs as KNNs in the rest of the paper. When we refer to exact KNNG and exact KNNs, we will mention them explicitly.

In this paper, we focus on in-memory solutions on high-dimensional dense vectors with the Euclidean distance as the distance measure, which is the most popular setting in the literature. This setting makes us focus on the comprehensive comparison on existing works. We would solve this problem with other settings, e.g., disk-based solutions, sparse data and other distance measures, in future works.

Algorithm 1 shows the framework of KNNG construction, adopted by many methods in the literature [6, 34, 36, 29]. This framework contains two steps, i.e., INIT and NBPG. The philosophy of this framework is that INIT geneartes an initial KNNG quickly, and then NBPG efficiently refines it, which follows the principle “a neighbor of my neighbor is also likely to be my neighbor” [9]. Note that we do not expect INIT costs a lot for high accuracy, because the refinement will be done in NBPG.

In this paper, we will introduce a few representative approaches for INIT and NBPG respectively. For each method, we also analyze its memory requirement and time complexity. For memory requirement, we only count the auxiliary data structures of each method, ignoring the common structures, i.e., the data $D$ and the KNNG $G$.

Proximity graphs (e.g., SW [21], HNSW [22], DPG [17] and NSG [10]) are the state-of-the-art methods for ANN search. Like KNNG, they treat each point $u$ as a graph node, but have distinct strategies to define $u$’s edges. The relationship between KNNG and proximity graph is twisted. On one hand, KNNG could be treated as a special type of proximity graph. Hence, KNNG is used for ANN search in recent benchmarks [2, 1, 17]. In those benchmarks, KNNG is usually denoted as KGraph, which is actually a KNNG construction method [9]. On the other hand, some proximity graphs such as DPG and NSG require an accurate KNNG as the premise, so that their construction cost is higher than KNNG. However, SW and HNSW are not built on top of KNNG and thus may be built faster than KNNG with adequate settings. As a new perspective, we show that SW and HNSW could also be used to build KNNG.

Moreover, we show ANN search on a proximity graph in Algorithm 2, which will be discussed more than once in this paper. Its core idea is similar to NBPG, i.e., “a neighbor of my neighbor is also likely to be my neighbor”. Let $H$ be a proximity graph and $q$ be a query. The search starts on specified or randomly-selected node(s) $ep$, which are first pushed into candidate set $pool$ (a sorted node list). Let $L=\max(k,efSearch)$ be the size of $pool$. In each iteration in Line 4-11, we find the first unexpanded node $u$ in $pool$ and then expand $u$ in Lines 6-10, denoted as $expand(q,u,H)$. This expansion treats each neighbor $v$ of $u$ on $H$ as a candidate to refine $pool$ in Line 8-10, which is denoted as $update(pool,v)$ . Once the first $efSearch$ nodes in $pool$ have been expanded, the search process terminates and returns the first $k$ nodes in $pool$. Obviously, $efSearch$ is key to the search performance. A large $efSearch$ increases both the cost and the accuracy.

Wang et al. [34] proposed a multiple random divide-and-conquer approach to build base approximate neighborhood graphs and then ensemble them together to obtain an initial KNNG. Their method, which we denote as Multiple Division, recursively partitions the data points in $D$ into two disjoint groups. To assign nearby points to the same group, the partitioning is done along the principal direction, which could be rapidly generated [34]. The data set $D$ is partitioned into two disjoint groups $D_{1}$ and $D_{2}$ (as illustrating in Figure 3), which are further split in a recursive manner until the group to be split contains at most $T_{div}$ points. Then a neighborhood subgraph for each finally-generated group is built in a brute-force manner, which costs $O(d*T_{div}^{2})$.

A single division as described above yields a base approximate neighborhood graph containing a series of isolated subgraphs, and it is unable to connect a point with its closer neighbors lying in different subgraphs. Thus, Multiple Division uses $L_{div}$ random divisions. Rather than computing the principal direction from the whole group, Multiple Division computes it over a random sample of the group. For each point $u\in D$, it has a set of KNNs from each division and thus totally $L_{div}$ sets of KNNs, from which the best KNNs are generated. We illustrate this process in Figure 4.

Cost Analysis: Since those $L_{div}$ divisions could be generated independently, the memory requirement is $O(n)$ to store the current division. For each division, the recursive partition costs $O(n*d*\log n)$ and the brute-force procedure on all finally-generated groups requires $O(n*d*T_{div})$. With $L_{div}$ divisions, the cost of Multiple Division is $O(L_{div}*n*d*(\log n+T_{div}))$. We can see that $L_{div}$ and $T_{div}$ are key to the cost of Multiple Division.

Zhang et al. [36] proposed a method, denoted as LSH KNNG, which uses the locality sensitive hashing (LSH) technique to divide $D$ into groups with equal size, and then builds a neighborhood graph on each group using the brute-force method. To enhance the accuracy, similar to Multiple Division, $L_{hash}$ divisions are created to build $L_{hash}$ base approximate neighborhood graphs, which are then combined to generate an initial KNNG of higher accuracy.

The details of such a division are illustrated in Figure 5. For each $d$-dimensional vector, LSH KNNG uses Anchor Graph Hashing (AGH) [18], an LSH technique, to embed it into a binary vector in $\{-1,1\}^{b}$. Then, the binary vector is transferred to a one-dimensional projection on a random vector in $\mathbb{R}^{b}$. All points in $D$ are sorted in the ascending order of their one-dimensional projections and then divided into equal-size disjoint groups accordingly. Let $T_{hash}$ be the size of such a group. Then a neighborhood graph is built on each group in the brute-force manner.

Cost Analysis: Like Multiple Division, each LSH division could be built independently. Hence, its memory requirement is caused by the binary codes and the division result. In total, its memory requirement is $O(n*b)$. In each division, generating binary codes costs $O(n*d*b)$ and the brute-force procedure on all groups costs $O(n*d*T_{hash})$. In total, the cost of LSH KNNG is $O(L_{hash}*n*d*(b+T_{hash}))$. We can see that the cost of LSH KNNG is sensitive to $L_{hash}$ and $T_{hash}$, since $b$ is usually a fixed value.

Tang et al. [29] proposed a method, denoted as LargeVis, to construct a KNNG for the purpose of visualizing large-scale and high-dimensional data. Like Multiple Division, LargeVis recursively partitions the data points in $D$ into two small groups, but the partitioning is along a random projection (RP) that connects two randomly-sampled points from the group to be split. The partitioning results are organized as a tree called RP tree [7]. Similar to Multiple Division, $L_{tree}$ RP trees will be built to enhance the accuracy. An initial KNNG is thus generated by conducting a KNN search procedure for each point on those trees. Each search process will be terminated after retrieving $L_{tree}*k$ candidates ²²2Summarized according to the public codes from the authors https://github.com/lferry007/LargeVis. .

Cost Analysis: Unlike Multiple Division and LSH KNNG, we have to store all $L_{tree}$ RP trees in the memory, since ANN search for each point is done on those trees simultaneously. Each tree requires $O(n*d)$ space, since each inner node stores a random projection vector. Hence, the total memory requirement is $O(L_{tree}*n*d)$. For each query, its main cost is to reach the leaf node from the root node in each tree and then verify $L_{tree}*k$ candidates. Hence, the total cost of LargeVis is $O(L_{tree}*n*d*(\log n+k))$. We can see that $L_{tree}$ and $k$ are key to the cost of LargeVis.

SW graph [21] and HNSW graph [22] are the state-of-the-art methods for ANN search [1, 17]. In the literature, neither technique (SW or HNSW) has been used for INIT. According to their working mechanism, it is trivial to extend them for INIT, while it needs a thorough comparison with other INIT methods in the literature.

The SW graph $G_{sw}$ construction process starts from an empty graph, and iteratively inserts each data point $u$ into $G_{sw}$. In each insertion, we first treat $u$ as a new node and then conduct ANN search $\mathcal{A}=Search\_on\_Graph(G_{sw},u,M_{sw},efConstruction,ep)$ as in Algorithm 2, where $G_{sw}$ is the current SW graph and contains only a part of $D$. Afterwards, undirected edges are created between $u$ and $M_{sw}$ neighbors in $\mathcal{A}$. Figure 6 illustrates the insertion process for a data point $u$. It is easy to generate an initial KNNG during the construction of $G_{sw}$. Before building $G_{sw}$, each point’s KNN set is initialized as empty. When inserting $u$, its similar candidates will be retrieved by the search process and a distance is computed between $u$ and each candidate $v$. We use $v$ (resp. $u$) to refine the KNN set of $u$ (resp. $v$). Once the SW graph is constructed, an initial KNNG is also generated. For simplicity, this method is denoted as SW KNNG.

Cost Analysis : The main memory requirement of SW KNNG is to store the SW graph, which needs $O(n*M_{sw})$ space. Its main cost is to insert $n$ nodes. By experiments, we find that each insertion expands $O(efConstruction)$ nodes, as demonstrated in Appendix A. But, there is no explicit upper bound on the neighborhood size. Let $M_{sw}^{max}$ ³³3$M_{sw}^{max}$ is specific w.r.t. the data set, which is obviously affected by the hubness phenomenon as discussed in Section 5.1 be maximum neighborhood size in $G_{sw}$. In the worst case, its time complexity is $O(n*d*efConstruction*M_{sw}^{max})$.

The HNSW garph $G_{hnsw}$ is a hierarchical structure of multiple layers. A higher layer contains fewer data points and layer 0 contains all data points in $D$. To insert a new point $u$, an integer maximum layer $l_{h}(u)$ is randomly selected so that $u$ will appear in layers 0 to $l_{h}(u)$. To insert $u$, the search process starts from the entry point $ep$ in the highest layer. The nearest node found in the current layer will serve as the entry point of search in the next layer, and this search process repeats in the next layer till layer 0. Note that the search in each layer follows Algorithm 2. After the search, HNSW creates undirected edges between $u$ and $M_{hnsw}$ neighbors selected from $efConstruction$ returned ones, which is done in layers from $l_{h}(u)$ to layer 0 respectively. Unlike SW, HNSW imposes a threshold $M_{hnsw}$ on the neighborhood size. Once the edges of a node exceeds $M_{hnsw}$, HNSW prunes them by a link diversification strategy. Figure 7 illustrates an insertion in HNSW. By the same process in SW KNNG, we build the initial KNNG by tracing each pair of distance computation when building the HNSW graph. We denote this method as HNSW KNNG.

Cost Analysis: The main requirement of HNSW KNNG is caused by the HNSW graph, which contains $O(n)$ nodes and each node has up to $M_{hnsw}$ neighbors. Hence, its memory requirement is $O(n*M_{hnsw})$. The cost of HNSW KNNG mainly contains two parts, i.e., ANN search for each node and pruning operations for oversize nodes. We find that each search expands $O(efConstruction)$ nodes as demonstrated in Appendix B and thus the total cost for ANN search is $O(n*d*M_{hnsw}*efConstruction)$. Each pruning operation costs $O(M_{hnsw}^{2}*d)$, but the exact number of those operations is unknown in advance. In the worst case, adding each undirected edge will lead to a pruning operation. In this case, there are totally $n*M_{hnsw}$ pruning operations. However, the practical number of pruning operations on real data is far smaller than $n*M_{hnsw}$, as shown in Appendix B. Overall, the total cost of HNSW KNNG is $O(n*d*M_{hnsw}*(efConstruction+M_{hnsw}^{2}))$ in the worst case.

As a popular NBPG method, UniProp stands for uni-directional propagation, which iteratively refines a node’s KNNs from the KNNs of its KNNs [36, 29]. Let $N_{k}^{t}(u)$ be the KNNs of $u$ found in iteration $t$. We can obtain $N_{k}^{0}(u)$ from the initial KNNG constructed in the INIT step. Algorithm 3 shows how to iteratively refine the KNNs of each node. In iteration $t$, we initially set $N_{k}^{t}(u)$ as $N_{k}^{t-1}(u)$ and then conduct $update(N_{k}^{t}(u),w)$ (as in Algorithm 2) for each $w\in\cup_{v\in N_{k}^{t-1}(u)}N_{k}^{t-1}(v)$. UniProp repeats for $nIter$ times and terminates.

Cost Analysis: UniProp needs to store the last-iteration KNNG. Hence its space requirement is $O(n*k)$. Its time complexity is $O(nIter*n*d*k^{2})$ in the worst case.

In contrast to UniProp, another type of method considers both the nearest neighbors and the reverse nearest neighbors of a node for neighborhood propagation  [9]. Thus we denote it as BiProp which stands for bi-directional propagation. For each node $u$, BiProp maintains a larger neighborhood $u.pool$ with size $L\geq k$ , which contains $u$’s $L$ nearest neighbors found so far. BiProp also takes an iterative strategy. Let $N_{m}^{t}(u)$ be the $m$ nearest neighbors of $u$ in iteration $t$, where $m$ is a variable and does not necessarily equal to $k$. Note that $N_{m}^{t}(u)\subset u.pool$. We use $R_{m}^{t}(u)$ to denote $u$’s reverse neighbor set $\{v|u\in N_{m}^{t}(v)\}$ and thus $\sum_{u\in D}|R_{m}^{t}(u)|=\sum_{v\in D}|N_{m}^{t}(v)|$. Then we denote the candidate set for neighborhood propagation as:

We show the procedure of BiProp in Algorithm 4. $u.pool$ is initialized randomly, and so is $B_{m}^{0}(u)$. In iteration $t$, a brute-force search is conducted on $B_{m}^{t-1}(u)$ for each $u$, as in Lines 4 to 9. At the end of each iteration, BiProp will update $N_{m}^{t}(u)$ and $B_{m}^{t}(u)$ in Lines 10 and 11.

In practice, the reverse neighbor set $R_{m}^{t}(u)$ may have a large cardinality (e.g., tens of thousands or even more) for some $u\in D$. Processing such a super node in one iteration costs as high as $O(d*(m+|R_{m}^{t}(u)|)^{2})$. To address this issue, BiProp imposes a threshold $T$ on $|R_{m}^{t}(u)|$ and the oversize sets will be reduced by random shuffle in each iteration.

Both NNDes¹³¹³13As a well-known KNNG construction method, it is publicly available on https://code.google.com/archive/p/nndes. and KGraph¹⁴¹⁴14It is publicly available on https://github.com/aaalgo/kgraph. It is well known as a method for ANN search, but ignored for KNNG construction. follows the idea of BiProp. Their key difference is how to set $m$ and $L$. In NNDes, $L=k$ and $m\leq k$ for all nodes, which is specified in advance. As in KGraph, $L$ is a specified value, but $m$ varies with node $u$ and the iteration $t$, denoted as $m^{t}(u)$. To determine $m^{t}(u)$, three rules must be satisfied simultaneously, summarized from the source codes. Let $\delta$ be a small value (e.g., 10).

• •

  Rule 1 : $m^{0}(u)=\delta$ and $m^{t}(u)-m^{t-1}(u)\leq\delta$.

• •

  Rule 2 : $|N_{m}^{t}(u)\setminus N_{m}^{t-1}(u)|\leq\delta$.

• •

  Rule 3 : $m^{t}(u)$ increases, iff $\exists v\in B_{m}^{t-1}(u)$ such that the brute-force search on $B_{m}^{t-1}(u)$ refines $N_{k}^{t}(v)$.

Cost Analysis: In NNDes, we need to store $N_{m}^{t}(\cdot)$ and $B_{m}^{t}(\cdot)$. Since $m\leq k$ and $T=k$, the required memory is $O(n*k)$. Its time complexity is $O(nIter*n*d*k^{2}$) in the worst case. In KGraph, we need to store $u.pool$ and $B_{m}^{t}(u)$ for each $u$. $|N_{m}^{t}(u)|$ is bounded by $L$ and thus $|B_{m}^{t}(u)|\leq L+T$. Its memory requirement is $O(n*L)$, which could be large enough since $L$ is usually set as a large value (e.g., 100). Its time complexity is $O(nIter*n*d*(L+T)^{2})$ in the worst case, but it is very fast in practice due to the filters as discussed in Section 4.4.

Wang et al. introduced the initial idea of DeepSearch [34], as shown in Algorithm 5. It follows the idea of ANN search on a proximity graph $H$, created online before neighborhood propagation. For $u\in D$, it refines $N_{k}(u)$ by conducting ANN search with staring nodes from the initial KNNG. Unlike UniProp and BiProp, DeepSearch runs only for one iteration.

The proximity graph $H$ is key to the performance of DeepSearch. An appropriate $H$ should satisfy two conditions. First, $H$ should be rapidly built, since it is built online. Second, $H$ should have good ANN search performance. $H$ is the initial KNNG built by Multiple Division in [34], denoted as DeepMdiv, which satisfies the first condition but fails for the second. Among existing proximity graphs, HNSW is a good choice of DeepSearch, which satisfies both conditions. We denote this method as Deep HNSW. Others such as DPG [17] and NSG [10] require an expensive modification on an online-built KNNG, which violates the first condition. Thus they are not suitable for DeepSearch. The construction cost of SW is unstable as shown in Section 3.5 and its ANN search performance is usually worse than other proximity graphs [17]. Hence SW is not suitable.

Cost Analysis: DeepSearch needs to store the proximity graph. Thus DeepMdiv requires $O(n*k)$ and Deep HNSW needs $O(n*M_{hnsw})$. The cost of DeepSearch is determined by two factors, i.e. the number of expanded nodes and their neighborhood sizes. We find that both DeepMdiv and Deep HNSW expand $O(efSearch)$ nodes, as demonstrated in Appendix C. Each node on KNNG has $k$ neighbors and thus the cost of DeepMdiv is $O(n*d*efSearch*k)$. Each node on the HNSW graph has at most $M_{hnsw}$ edges and thus the cost of Deep HNSW is $O(n*d*efSearch*M_{hnsw})$.

In NBPG, there exist repeated distance calculations for the same pair of points, which increases the computational cost. In this section, we discuss how to reduce such repeated computations which can be categorized into two sources: intra-iteration and inter-iteration.

In the left graph in Figure 9, $w_{1}$ is a neighbor of both $v_{1}$ and $v_{3}$, which are the neighbors of $u$. In the neighborhood propagation, the pair $(u,w_{1})$ will be checked twice in the same iteration. Thus we call this case intra-iteration repetition. To avoid the intra-iteration repetition, we use a global filter, a bitmap $visited$ of size $n$ in each iteration. $visited[u]$ indicates whether $u$ has been visited or not. Let us take UniProp as an example. Before line 4 in Algorithm 3, we initialize all $visited$ elements as $false$. Then we check $visited[w]$ for candidate $w$ before line 6. If $w$ is a new candidate, we conduct $update(N_{k}^{t}(u),w)$ and mark $visited[w]$ as $true$. Otherwise, $w$ is just ignored. The global filter is also applicable to BiProp and DeepSearch.

Inter-iteration repetitions refer to the repeated computation in different iterations. Consider the two iterations $t_{1},t_{2}$ in Figure 9 in which the pair $(u,w_{2})$ is considered twice. To avoid the inter-iteration repetition, we can use a local filter. Let us take UniProp as an example. For each neighbor $v\in N_{k}^{t}(u)$, we assign it an attribute $isOld$, which is set as $false$ at start. If $v$ is firstly added to $N_{k}^{t}(u)$ as a new neighbor, its $isOld$ is $false$, otherwise it is $true$ as an old neighbor. In Figure 9, we use the solid arrows to represent new KNN pairs and the dashed arrows for old KNN pairs. In iteration $t_{2}$, $w_{2}$ is an old neighbor of $v_{3}$ and $v_{3}$ is an old neighbor of $u$. Thus the pair $(u,w_{2})$ has been considered in a previous iteration and $w_{2}$ should be ignored by $u$. Note that the inter-iteration repetition cannot be completely avoided by the local filter. Consider $u$ finds $w_{1}$ via a new neighbor $v_{2}$ in iteration $t_{2}$, which is a repeated computation from iteration $t_{1}$. The local filter can also be used for BiProp, but is not necessary for DeepSearch with only one iteration.

According to our knowledge, the global filter has been widely used in existing studies, while the local filter has not. The filters will accelerate NBPG methods, but will not change the cost analysis in the worst case. On the other hand, they introduce new data structures and require more memory. The space required by the global filter is $O(n)$, while that by the local filter is determined by the neighborhood sizes. To be specific, UniProp and NNDes require $O(n*k)$, while KGraph needs $O(n*L)$. Since the local filter is attached to the neighbors, it will not change the final space complexity.

We use the same experimental settings in Section 3.5.1.

According to [24], a hub node appears in the KNNs of many nodes, making it a popular exact KNN of other nodes. Accordingly, a hub node has a large in-degree in the exact KNNG. We show its definition from [24] as follows.

According to [24], a hub node is usually close to the data center or the cluster center, and thus it is naturally attractive to other nodes as one of their exact KNNs. Further, we extend the hubness concept to the whole data set as follows.

We can see that $H_{k}(x,D)$ is actually the normalized sum of the largest $\lceil x\times n\rceil$ $h_{k}$ values and thus $H_{k}(x,D)$ is between 0 and 1. For the eight data sets, we divide them into three groups according to their data hubness, i.e., low hub, moderate hub and high hub respectively, as shown in Table IV. We can see that Msdrp has the largest data hubness, where the first $0.1\%$ (995) points have $25.9\%$ hubness (corresponding to over 5 million edges in the exact KNNG).

In addition, we define the accuracy of reverse KNNs. For a KNNG $G$, we have $R_{k}(v)=\{u|v\in N_{k}(u)\}$ as $v$’s reverse KNNs. Then we define the accuracy of $v$’s reverse KNNs as $recall^{R}(v)=|R_{k}(v)\cap R_{k}^{*}(v)|/|R_{k}^{*}(v)|$, where $R_{k}^{*}(v)$ is defined as in Definition 2. $recall^{R}$ reflects the accuracy of a KNNG to some extent, since a node probably has a high $recall^{R}$ value on an accurate KNNG.

In this part, we present the effect of data hubness on the performance of NBPG. We discuss the performance in two aspects, i.e., (1) computational cost vs accuracy and (2) the converged accuracy. We use $scan\_rate=\frac{\#total\_dist}{n*(n-1)/2}$ to measure the computational cost, where $\#total\_dist$ indicates the total number of distance computations during KNNG construction and $n*(n-1)/2$ is the number of all distance computations in the brute-force method. Unlike $cost$, $scan\_rate$ is independent of $d$. Here, we show the results on data size in Figure 14. We select data sets with the same size in each subfigure, so that we can focus on the effects of data hubness. We can see that the data hubness has a significant effect on the performance of NBPG methods. With a larger data hubess, each NBPG method has to compute more distances to achieve the same $recall$.

The converged $recall$ is such a value that once an NBPG method reaches it, paying more cost cannot improve the $recall$ value in a significant scale. We believe that the converged $recall$ reflects the characteristic of an NBPG method. Here we obtain this value of a method by setting its parameters large enough. To be specific, the parameter settings are $nIter=4$ in LargeVis, $efSearch=160$ in Deep HNSW and $nIter=16$ in KGraph. We show the results in Table 3. We can see that a lower hub data usually has a higher converged $recall$, while a higher hub data has a lower converged $recall$. This indicates that it is computationally expensive to get a high-recall KNNG for a high hub data.

As a representative UniProp method, LargeVis starts from an initial KNNG with low or moderate accuracy and then conducts UniProp for a few iterations. Here we vary the number $nIter$ of iterations from 0 to 3. When $nIter=0$, it corresponds to the initial KNNG. Due to the space limit, we only report the results on Sift and Glove in Figure 15. The $x$-axis shows the node hubness of individual nodes and the $y$-axis shows the $recall$ or $recall^{R}$. We can see the $recall$ values of all nodes increases substantially, especially in the first iteration. Notably, those newly found exact KNNs are mostly the hub nodes, as reflected by the significant increase in the $recall^{R}$ values of hub nodes.