CO-SNE: Dimensionality Reduction and Visualization for Hyperbolic Data

A hyperbolic space is a Riemannian manifold with a constant negative curvature. There are several isometrically equivalent models for representing hyperbolic space. The Poincaré ball model is the most commonly used one in hyperbolic representation learning [23, 7]. The $n$-dimensional Poincaré ball model is defined as $(\mathbb{B}^{n},\mathfrak{g}_{\mathbf{x}})$, where $\mathbb{B}^{n}$ = $\{\mathbf{x}\in\mathbb{R}^{n}:\lVert\mathbf{x}\rVert<1\}$ and $\mathfrak{g}_{\mathbf{x}}=(\gamma_{\mathbf{x}})^{2}I_{n}$ is the Riemannian metric tensor. $\gamma_{\mathbf{x}}=\frac{2}{1-\lVert\mathbf{x}\rVert^{2}}$ is the conformal factor and $I_{n}$ is the Euclidean metric tensor. Given two points $\bm{u}\in\mathbb{B}^{n}$ and $\bm{v}\in\mathbb{B}^{n}$, the hyperbolic distance between them is defined as,

$$d_{\mathbb{B}^{n}}(\bm{u},\bm{v})=\operatorname{arcosh}\left(1+2\frac{\lVert\bm{u}-\bm{v}\rVert^{2}}{(1-\lVert\bm{u}\rVert^{2})(1-\lVert\bm{v}\rVert^{2})}\right)$$

(1)

where $\operatorname{arcosh}$ is the inverse hyperbolic cosine function and $\lVert\cdot\rVert$ is the usual Euclidean norm. Different from Euclidean distance, hyperbolic distance grows exponentially fast as we move the points towards the boundary of the Poincaré ball as in Figure 2.

t-SNE [31] begins by mapping high-dimensional distances between datapoints to similarity values. The similarity values are either conditional or joint probabilities based on the probability a point will pick another as its neighbor if neighbors are picked in proportion to the probability density of a distribution centered at that point. t-SNE defines the conditional probability $p_{j|i}$, the probability that the point $\mathbf{x}_{i}$ will pick a point $\mathbf{x}_{j}$ as its neighbor, using a normal distribution centered at the point $\mathbf{x}_{i}$. t-SNE then defines the joint probability distribution $P$, by setting $p_{ij}=\frac{p_{i|j}+p_{j|i}}{2m}$ as a way to increase the cost contribution of outlier points, where $m$ is the number of high-dimensional datapoints. The conditional probability density $p_{j|i}$ is,

$$p_{j|i}=\frac{\exp(-d(\mathbf{x}_{i},\mathbf{x}_{j})^{2}/2\sigma_{i}^{2})}{\sum_{k\neq i}\exp(-d(\mathbf{x}_{i},\mathbf{x}_{k})^{2}/2\sigma_{i}^{2})}$$

(2)

where $d(\mathbf{x}_{i},\mathbf{x}_{j})$ is the distance between $\mathbf{x}_{i}$ and $\mathbf{x}_{j}$. In the low-dimensional space, Student’s t-distribution is used for modeling the joint probability distribution $Q$ between embeddings, and $q_{ij}$ is defined as,

$$q_{ij}=\frac{(1+d(\mathbf{y}_{i},\mathbf{y}_{j})^{2})^{-1}}{\sum_{k\neq l}(1+d(\mathbf{y}_{k},\mathbf{y}_{l})^{2})^{-1}}$$

(3)

where $\mathbf{y}_{i}$ is the corresponding low-dimensional embedding of $\mathbf{x}_{i}$. In t-SNE, to maintain the local similarities, the cost function to minimize is the Kullback-Leibler divergence between the probability distributions $P$ and $Q$:

$$\mathcal{C}=KL(P||Q)=\sum_{i}\sum_{j}p_{ij}\log{\frac{p_{ij}}{q_{ij}}}$$

(4)

One direct extension of t-SNE is to replace Euclidean version of normal distribution and Student’s t-distribution with hyperbolic normal distribution and hyperbolic Student’s t-distribution. We call such a direct extension as HT-SNE. In Section 3.3 and Section 3.4, we show how to generalize the distributions to hyperbolic space.

To define the conditional probability in the high-dimensional hyperbolic space, we need to generalize the normal distribution to hyperbolic space. One natural generalization is called the Riemannian normal distribution which is the maximum entropy probability distribution given an expectation and a variance [26]. Given the Fréchet mean $\bm{\mu}\in\mathbb{B}^{n}_{c}$ and a dispersion parameter $\sigma>0$, the Riemannian normal distribution is defined as,

$$\mathcal{N}_{\mathbb{B}^{n}}(\bm{x}|\bm{\mu},\sigma^{2})=\frac{1}{\bm{Z}}\exp(-\frac{d_{\mathbb{B}^{n}}(\bm{\mu},\bm{x})^{2}}{2\sigma^{2}})$$

(5)

where $\bm{Z}$ is the normalization constant. There are other generalizations of the normal distribution in hyperbolic space [18]. We use the Riemannian normal distribution for simplicity. Thereafter, we refer to Riemannian normal distribution as hyperbolic normal distribution.

One way to define the Student’s t-distribution is to express the random variable $t$ as,

$$t=\frac{u}{\sqrt{v/n}}$$

(6)

where $u$ is a random variable sampled from a standard normal distribution and $v$ is a random variable sampled from a $\chi^{2}$-distribution of $n$ degrees of freedom. In particular, t-SNE adopts a Student’s t-distribution with one degree of freedom and the probability density function is defined as,

$$f(t;t_{0})=\frac{1}{\pi(1+(t-t_{0})^{2})}$$

(7)

To extend the Student’s t-distribution to hyperbolic space, we derive the probability density function as,

$$f_{\mathbb{B}^{n}}(t;t_{0})=\frac{1}{\pi(1+d_{\mathbb{B}^{n}}(t,t_{0})^{2})}$$

(8)

The details can be found in the Supplementary.

The motivation to use Student’s t-distribution in the standard t-SNE is that Student’s t-distribution has heavier tails than the normal distribution. This causes a repulsion force between dissimilar high-dimensional datapoints which helps mitigate the “Crowding Problem” [31]. However, hyperbolic Student’s t-distribution is not heavy-tailed since the hyperbolic distance grows exponentially fast.

Figure 3 plots the hyperbolic normal distribution and hyperbolic Student’s t-distribution. It can be noted that the tails of hyperbolic Student’s t-distribution diminish as fast as hyperbolic normal distribution. In contrast, the standard Euclidean Student’s t-distribution has heavier tails than the normal distribution. The consequence is that the repulsion force and the attraction force with hyperbolic Student’s t-distribution behave drastically different from that of using Euclidean Student’s t-distribution.

Similar to [31], we plot the gradients between two low-dimensional embeddings $\mathbf{y}_{i}$ and $\mathbf{y}_{j}$ as a function of their pairwise Euclidean (hyperbolic) distance in the low-dimensional Euclidean (hyperbolic) and the pairwise Euclidean (hyperbolic) distance of the corresponding datapoints in the high-dimensional Euclidean (hyperbolic) space. Positive values indicate attraction and negative values indicate repulsion between the embeddings $\mathbf{y}_{i}$ and $\mathbf{y}_{j}$ respectively. Here we only consider the initial stage when all the low-dimensional embeddings are close. The results are shown in Figure 3. We have two observations. First, for the standard t-SNE, there is strong repulsion when dissimilar high-dimensional datapoints are projected closely. Second, with the hyperbolic Student’s t-distribution in HT-SNE, there is little or no repulsion force which causes the low-dimensional embeddings tend to cluster together.

To remedy this issue, we seek to replace the hyperbolic Student’s t-distribution to create more repulsion forces between dissimilar high-dimensional datapoints. Consider the hyperbolic Cauchy distribution which has the probability density function,

$$f(t;t_{0},\gamma)=\frac{1}{\pi\gamma}[\frac{\gamma^{2}}{d_{\mathbb{B}^{n}}(t,t_{0})^{2}+\gamma^{2}}]$$

(9)

where $\gamma$ is the scale parameter. Notice that the Student’s t-distribution is a special case of Cauchy distribution with $\gamma=1.0$. With a small $\gamma$, Cauchy distribution has a higher peak (Figure 3 (a)). Thus, if the high-dimensional datapoints are modeled with close low-dimensional embeddings, the density value is much larger than the corresponding density in the high-dimensional space. This produces a strong repulsion when dissimilar high-dimensional datapoints are modeled with close low-dimensional embeddings. More details can be found in the Supplementary. We show the corresponding gradients in Figure 3 (d). We can observe that there is a strong repulsion force when high-dimensional dissimilar datapoints are projected closely in CO-SNE. Such repulsion is particularly important in the initial stage of training since all the low-dimensional embeddings are close.

Hyperbolic space can be naturally used for embedding tree structured data. The datapoint which is close to the center of the Poincaré ball can be viewed as the root node and datapoints which are close to the boundary of the Poincaré ball can be viewed as leaf nodes. The criterion of t-SNE cannot preserve the tree-structure of the high-dimensional datapoints in the low-dimensional space. One natural way to characterize the level of the datapoint in the underlying tree is the norm of the datapoint. Thus, we attempt to keep the norm invariant after the high-dimensional datapoint is projected. This leads to the following loss function which minimizes the difference between the norms of the high-dimensional datapoint $\mathbf{x}_{i}$ and the corresponding low-dimensional embedding $\mathbf{y}_{i}$,

$$\mathcal{H}=\frac{1}{m}\sum_{i=1}^{m}(\lVert\mathbf{x}_{i}\rVert^{2}-\lVert\mathbf{y}_{i}\rVert^{2})^{2}$$

(10)

With the distance loss, we can preserve the global hierarchy of the high-dimensional hyperbolic embeddings.

We first use a synthetic dataset to validate the efficacy of CO-SNE. We randomly generate five point clusters of 20 points each in the 5D hyperbolic space. Each follows a hyperbolic normal distribution with the unit variance and the mean located on a different axis. The first and second means are close to the origin, at [0.1, 0, 0, 0, 0] and [0, -0.2, 0, 0, 0] respectively, whereas the third and fourth means are far from and equi-distance to the origin, at [0, 0, 0.9, 0, 0] and [0, 0, 0, -0.9, 0] respectively. The last mean is right at the origin [0, 0, 0, 0, 0]. Figure 4 shows the 2D visualization results of these 5D points by different methods.

CO-SNE produces a much better visualization compared with the baselines. With CO-SNE, the projected two-dimensional hyperbolic embeddings can preserve the local similarity structure and the global hierarchical structure of high-dimensional datapoints well. Also, CO-SNE can prevent the high-dimensional datapoints from being projected too close which usually happens with Euclidean distance based methods. Note that HT-SNE does not have not enough repulsion to push close low-dimensional embeddings apart. We provide a detailed analysis on the drawbacks of each baseline for visualizing high-dimensional hyperbolic datapoints.

• •

  Standard t-SNE: Euclidean distances are used for computing similarities in the high-dimensional space. Hyperbolic distances grow much faster than Euclidean distances. For high-dimensional hyperbolic datapoints which are close to boundary of the Poincaré ball, the standard t-SNE wrongly underestimates the distance between them. As a consequence, the standard t-SNE would take dissimilar high-dimensional data points as neighbors. The resulting low-dimensional embeddings would collapse into one point which leads to poor visualization. In summary, t-SNE cannot preserve the global hierarchy of the hyperbolic data.

• •

  PCA and HoroPCA: as mentioned above, PCA and HoroPCA are linear dimensionality reduction methods which are not generally suitable for visualization in a two-dimensional space. Both PCA and HoroPCA cannot preserve local similarity of the hyperbolic data.

• •

  UMAP: UMAP suffers from the same issue as t-SNE since Euclidean distance is used for computing high-dimensional similarities.

We mainly compare CO-SNE with HoroPCA since HoroPCA is specifically designed for hyperbolic data.

Biological data can reveal naturally occurring hierarchies, such as in single cell RNA sequencing data [15]. In [15], they analyze cellular differentiation data, the transition of immature cells to specialized types. The immature cells can be viewed as the root of the tree and can branch off into several different types of cells, creating hierarchical data of cells in different states of progress in the transition process. One dataset we adapt from [15] is the mouse myelopoiesis dataset presented by [24], where there are 532 cells of 9 types. Two of the types, HSPC-1 and HSPC-2, form the root of the hierarchy, while megakaryocytic (Meg), erythrocytic (Eryth), monocyte-dendritic cell precursor (MDP), monocytic (Mono), and myelocyte (myelocytes and metamyelocytes) type cells are states father from the root. Granulocytic (Gran) cells are a precursor to myelocytes and multi-lineage primed (Multi-Lin) cells are in an intermediate state.

The data has originally 382 dimensions (noisy) and like [15] we first reduce it to 20 dimensions via PCA to reduce the noises. We then scale the data to fit within the Poincaré ball and run CO-SNE and HoroPCA to produce two-dimensional hyperbolic embeddings. Noted that this dataset is not centered. The results are shown in Figure 5. The proposed low-embeddings by CO-SNE capture the hierarchical structure in the original data.

Hyperbolic space has been used to embed hierarchical representations of symbolic data. In [23], the authors adopt hyperbolic space for embedding taxonomies, in particular, the transitive closure of WordNet noun hierarchy [21]. As shown in [23], higher-dimensional hyperbolic embeddings often lead to better representations, but they are harder to visualize. Following [23], we embed the hypernymy relations of the mammals subtree of WordNet in hyperbolic space. We use the open source implementation provided by [23] to train the ten-dimensional embeddings. We use HoroPCA and CO-SNE to visualize the learned embeddings in a two-dimensional hyperbolic space.

Figure 6 shows that compared with HoroPCA, CO-SNE can better preserve the hierarchical and similarity structure of the high-dimensional datapoints. For example, the word $feline$ and $canine$ are close to $canivore$ in the CO-SNE embeddings, which is not the case in HoroPCA. The embeddings produced by CO-SNE also more resemble the two-dimensional embeddings as shown in Figure 2b of [23].

We apply CO-SNE to visualize the embeddings produced by hyperbolic neural networks for supervised image classification. We train a hyperbolic neural network (HNN) [7] with feature clipping [9] on MNIST. The clipping value is 1.0 and the feature dimension is 64. We use HoroPCA and CO-SNE to reduce the dimensionality of the features to two. The full test set of MNIST cannot be used due to the out-of-memory issue of HoroPCA. So for each class, we randomly sample 100 images.

Figure 7 shows the two-dimensional embeddings generated by HoroPCA and CO-SNE. The visualization produced by CO-SNE is significantly better the visualization produced by HoroPCA. In CO-SNE, the classes are well separated and have a clear hierarchical structure. We further train hyperbolic classifiers [7] on the frozen two-dimensional features. We use a learning rate of 0.001 and the number of epoch is 10. The accuracy of the features generated by HoroPCA is 30.2% while the accuracy of the features generated by CO-SNE is 61.2%. This implies that low-dimensional features generated by CO-SNE are more separated and respect the structure of original high-dimensional embeddings.

CO-SNE can be used to visualize hyperbolic datapoints in a two-dimensional hyperbolic space while maintaining the local similarities and hierarchical structure. CO-SNE still suffers from the same weaknesses as t-SNE. In particular, as a general method for dimensionality reduction, the curse of intrinsic dimensionality and the non-convexity, please see [31] for more details. More ablations and results can be found in the Supplementary. In particular, we investigate the effect of hyperparameters for balancing the KL-divergence and the distance loss.

We give a brief overview of Riemannian geometry, for more details please refer to [4]. A Riemannian manifold $(\mathcal{M},\mathfrak{g})$ is a real smooth manifold $\mathcal{M}$ with a Riemannian metric $\mathfrak{g}$. The Riemannian metric $\mathfrak{g}$ is a smoothly varying inner product which is defined on the tangent space $T_{\mathbf{x}}{\mathcal{M}}$ of $\mathcal{M}$. Given $\mathbf{x}\in\mathcal{M}$ and two vectors $\mathbf{v},\mathbf{w}\in T_{\mathbf{x}}{\mathcal{M}}$, we can use the Riemannian metric $\mathfrak{g}$ to compute the inner product $\langle\mathbf{v},\mathbf{w}\rangle_{\mathbf{x}}$ as $\mathfrak{g}(\mathbf{v},\mathbf{w})$. The norm of $\mathbf{v}\in T_{\mathbf{x}}{\mathcal{M}}$ is defined as $\lVert\mathbf{v}\rVert_{\mathbf{x}}=\sqrt{\langle\mathbf{v},\mathbf{v}\rangle}_{\mathbf{x}}$. A geodesic generalizes the notion of straight line in the manifold which is defined as a curve $\gamma:[0,1]\rightarrow\mathcal{M}$ of constant speed that is everywhere locally a distance minimizer. The exponential map and the inverse exponential map are defined as follows: given $\mathbf{x},\mathbf{y}\in\mathcal{M},\mathbf{v}\in T_{\mathbf{x}}{\mathcal{M}}$, and a geodesic $\gamma$ of length $\lVert\mathbf{v}\rVert$ such that $\gamma(0)=\mathbf{x},\gamma(1)=\mathbf{y},\gamma^{\prime}(0)=\mathbf{v}/\lVert\mathbf{v}\rVert$, the exponential map $\textnormal{Exp}_{\mathbf{x}}:T_{\mathbf{x}}{\mathcal{M}}\rightarrow\mathcal{M}$ satisfies $\textnormal{Exp}_{\mathbf{x}}(\mathbf{v})=\mathbf{y}$ and the inverse exponential map $\textnormal{Exp}^{-1}_{\mathbf{x}}:\mathcal{M}\rightarrow T_{\mathbf{x}}{\mathcal{M}}$ satisfies $\textnormal{Exp}^{-1}_{\mathbf{x}}(\mathbf{y})=\mathbf{v}$.

Recall the way to define the the Student’s t-distribution which expresses the random variable $t$ as,

$$t=\frac{u}{\sqrt{v/n}}$$

(15)

where $u$ is a random variable sampled from a standard normal distribution and $v$ is a random variable sampled from a $\chi^{2}$-distribution of $n$ degrees of freedom. The $\chi^{2}$-distribution can also be derived from normal distribution. Let $u_{1}$, $u_{2}$, …, $u_{n}$ be independent standard normal random variables, then the sum of the squares,

$$v=\sum_{i=1}^{n}u_{i}^{2}$$

(16)

is a $\chi^{2}$-distribution with $n$ degrees of freedom. Thus, the probability density function of the $\chi^{2}$-distribution can be derived from the probability density function of the normal distribution which is,

$$T_{n}(v)=\frac{1}{2^{n/2}\Gamma(n/2)}v^{(n-2)/2}e^{-v/2}$$

(17)

Using Equation 15, we can further derive the probability density function of the Student’s t-distribution,

$$f_{n}(t)=\frac{1}{\sqrt{n}B(1/2,n/2)}(1+\frac{t^{2}}{n})^{-(n+1)/2}$$

(18)

where $B$ is the Beta function.

The probability density function of hyperbolic Cauchy distribution can be derived in a similar way using hyperbolic normal distribution.

Similar to the Student’s t-Distribution, the probability density function of Cauchy distribution can derived from the probability density function of the normal distribution. In particular, let $X$ and $Y$ be independent standard normal random variables, then $Z=\frac{X}{X+Y}$ is a Cauchy random variable. The probability density function of Cauchy distribution can be written as,

$$f(x;x_{0},\gamma)=\frac{1}{\pi\gamma[1+(\frac{x-x_{0}}{\gamma})^{2}]}$$

(19)

The probability density function of hyperbolic Cauchy distribution can be derived in a similar way using hyperbolic normal distribution.

The repulsion and attraction forces in CO-SNE depend on the term $p_{ij}-q_{ij}$ in Equation 12 of the main text. $p_{ij}$ is fixed during training which depends on the distribution of the high-dimensional datapoints. To create more repulsion forces between two close low-dimensional embeddings $y_{i}$ and $y_{j}$, we aim at increasing the probability that the point $y_{i}$ would select the point $y_{j}$ as its neighbor (i.e., $q_{ij}$). By using a small $\gamma$ in hyperbolic Cauchy distribution, the distance between $y_{i}$ and $y_{j}$ is scaled up. When the point $y_{i}$ is fixed, the probability of selecting $y_{j}$ as a neighbor (i.e., $q_{ij}$) is scaled up relatively to some point $y_{k}$ which is far away from $y_{i}$. As a result, $p_{ij}-q_{ij}$ can potentially become negative. This creates additional repulsion forces to push the low-dimensional points apart.

Recall the objective function of CO-SNE,

$$\mathcal{L}=\lambda_{1}\mathcal{C}+\lambda_{2}\mathcal{H}$$

(20)

$\lambda_{1}$ and $\lambda_{2}$ are used to balance the KL-divergence $\mathcal{C}$ and the distance loss $\mathcal{H}$ and can be regarded as the learning rates. For ablation studies on the effect of $\lambda_{1}$ and $\lambda_{2}$, we reuse the settings in Section 4.1 of the main text.

We consider different settings of $\lambda_{1}$ and $\lambda_{2}$ to investigate the effect of the KL-divergence and the distance loss. The results are shown in Figure 9. We have several observations from the results.

1. 1.

   In the first row, $\lambda_{1}=0.0$. This means that only the distance loss is presented. We can observe that the low-dimensional embeddings can only approximate the magnitude of the high-dimensional datapoints but not the similarity structure.

2. 2.

   In the first column, $\lambda_{2}=0.0$. This means that only the KL-divergence is presented. We can observe that the low-dimensional embeddings can only preserve the similarity structure in the high-dimensional datapoints but not the hierarchical information.

3. 3.

   In other cases, we can observe that a larger $\lambda_{2}$ can better preserve the hierarchical structure in the high-dimensional datapoints but may distort the similarity structure. A larger $\lambda_{1}$ may also lead to a bad visualization of the similarity structure since the KL-divergence might diverge. Nevertheless, CO-SNE is robust to wide choices of $\lambda_{1}$ and $\lambda_{2}$. Both the KL-divergence and the distance loss are important for producing good visualization.