Supplemental material for Visualizing theory space: Isometric embedding of probabilistic predictions, from the Ising model to the cosmic microwave background

The anisotropy in CMB radiation can be characterized by a $2\times 2$ direction dependent intensity matrix $I_{ij}(\hat{n})$ whose components can be recognized as 3 of the 4 Stokes parameters, I, Q, and U. The Q and U polarization maps can be made independent of the Stokes parameter measurement basis by separating them into divergence (E) and curl (B) components, to generate three maps of cosmological interest; the temperature fluctuation map T and two polarization maps, E and B. These can be expanded into spherical harmonics,

The anisotropies are expected to be (approximately) Gaussian. All of the Gaussian information can be extracted form the angular power spectra, which are defined defined as the cross correlation of the coefficients in the expansion and written as

Using this, we can construct a correlation matrix for the fluctuations,

The values of $C_{\ell}$ depend on the $\Lambda$CDM parameters, and likelihood analyses of CMB data fit with such a correlation function have been extensively studied, as they are invaluable for fitting and forecasting CMB measurements (e.g. collaboration (2015); Hamimeche and Lewis ; Tegmark (1997); Abazajian et al. (2016)). The probability of a fit for this data can be expressed as

This conditional probability defines the likelihood (collaboration, 2015; Tegmark, 1997), $\mathcal{L}(\{\hat{a}_{lm}\}|{\mbox{\boldmath$\theta$}})=p(\{\hat{a}_{lm}\}|{\mbox{\boldmath$\theta$}})$. The metric is given by the Fisher Information Matrix (FIM),

We can evaluate this integral by looking at the second derivatives of $\mathcal{L}$;

This expansion can be combined with Eq. S5 to extract all terms independent of the data. Thus, the first two terms in the sum can be completely pulled out of the integral. The remaining term is harder, and to evaluate it we make use of the following integral for symmetric, positive definite $M\times M$ matrix $A$ and symmetric $M\times M$ matrix $B$

This allows us to solve Eq. S5, setting $A=C_{\ell}^{-1}$ and $B=\partial_{\alpha}\partial_{\beta}C_{\ell}^{-1}$. We can now combine all the pieces together, and obtain a formula for the FIM

We can compare this to to previous results for FIM derivations, Perotto et al. (2006); Tegmark (1997) and confirm that we obtain the same result. We can decompose this as a sum over $\ell$ and the different spectra to obtain:

where $J^{\ell}_{XY,\mu}=\partial C^{XY}_{\ell}/\partial\theta_{\mu}$ is a tensor of partial derivatives, and $\Omega$ is given by Eq. S8 where derivatives are taken with respect to the $C_{\ell}^{XY}$. We can express $\Omega$ as a block diagonal matrix because the $C_{\ell}^{XY}$ are uncorrelated for different values of $\ell$, and in such a form represents the FIM where the parameters of interest are the $C_{\ell}^{XY}$. Note that $g_{\mu\nu}$ is not constant in the coordinates given by $C_{\ell}$.

For regular least-squares fitting, $\Omega$ would simply be a constant matrix representing experimental uncertainty. However, for CMB spectra, it is parameter dependent; it varies with the $C_{\ell}$. Geometrically, this can be interpreted as the metric in the $C_{\ell}$ embedding space. Since it varies with the $C_{\ell}$, it produces a non-Euclidean embedding. Visualizing the model manifold in this space is therefore problematic, since the space is warped and distorted, and distances are not faithfully represented as show in Fig. S1 a. This problem can be solved if we instead consider the probability distributions the $C_{\ell}$ correspond to for different parameters. In Table S1 we present the range of $\Lambda$CDM parameters explored in our model manifold. In for following section we explore these probability distributions for different $C_{\ell}$.

The set of probability distributions from a model generates a ‘probability simplex’, since they are all normalized to one. The Fisher Information Matrix in this space is non-Euclidean; it is diagonal with entries that are parameter dependent. For instance, in this space the Fisher Information for the Ising model is given by

This is a non-Euclidean metric, since it is not proportional to $\delta_{{\mathbf{S}}_{\mu},{\mathbf{S}}_{\nu}}$. It has a parameter dependent component given by the Boltzmann likelihood of being in a given spin state. This is similar to the manifold illustrated in Fig. S1 a, which shows the non-Euclidean embedding of CMB spectra in a space whose metric is also point-dependent and detailed in Section I.

If instead we consider the square root of a normalized probability distribution of a fit to data ${\mathbf{x}}$, $2\sqrt{{\cal{L}}({\mathbf{x}}|{\mbox{\boldmath$\theta$}})}$, then we can generate a model manifold embedded on an $n$-sphere of radius two, such that the metric is the Fisher Information Metric. Since ${\cal{L}}({\mathbf{x}}|{\mbox{\boldmath$\theta$}})$ is normalized to one and always positive, the dot product between $\sqrt{{\cal{L}}({\mathbf{x}}|{\mbox{\boldmath$\theta$}}_{1})}$ and $\sqrt{{\cal{L}}({\mathbf{x}}|{\mbox{\boldmath$\theta$}}_{2})}$ must be less than one, or equal to one if the likelihood functions are the same. The distance between two points on this $n$-sphere is proportional to the Hellinger distance Hellinger (1909), an f-divergence similar to the Kullback-Leibler divergence. It is straightforward to show that the metric for this $n$-sphere of radius two embedding is given by the Fisher Information Matrix, by considering the distance for some small perturbation $\delta{\mbox{\boldmath$\theta$}}$:

This embedding is therefore isometric, preserving distances as given by the Hellinger divergence Hellinger (1909). Unfortunately, there is a maximum distance any two points can be in this embedding, and that is given by the points on the poles attached to the positive orthant of the $n$-sphere. These are a distance $\sqrt{8}$ apart if the radius is two. Therefore, as more and more data are collected, creating increasingly orthogonal points, the manifold ‘winds around’ the $n$-sphere. The image generated by this is not ‘faithful’, in the sense that it does not allow for low-dimensional representations, as shown in Fig. S1 c.

Distances between predictions for two parameter combinations ${\mbox{\boldmath$\theta$}}_{1}$ and ${\mbox{\boldmath$\theta$}}_{2}$ in our intensive embedding are given by:

To determine the metric for this embedding, we consider a small parameter perturbation $\delta{\mbox{\boldmath$\theta$}}$ around some parameter combination $\theta$:

producing the same fisher information metric shown in Eq. II. For simplicity, we have dropped all terms from the expansion equal to zero. By preserving the local metric, our intensive embedding is isometric.

In order to visualize the various model manifolds in Fig. S1, we performed a principal component analysis on the data. This process rotates the data into an orthogonal basis such that the first component is along the direction of greatest variation, the second direction is along the direction of second greatest variation, and so on. In order to accomplish this, a data set $\{{\mathbf{d}}^{(J)}\}$ is produced, indexed by the superscript $(J)$. A data matrix can be produced, $D_{iJ}=d^{(J)}_{i}$, and the index $i$ indicates the vector component of the data. This can be discrete, such as the probability state vector for the Ising model, or continuous such as the likelihood of observing a certain fluctuation $a_{lm}$ in a CMB map.

The columns of $D$ are centered, producing a matrix $M_{iJ}=d^{(J)}_{i}-\tilde{d}_{i}$ where $\tilde{{\mathbf{d}}}=\frac{1}{n}\sum_{J}{\mathbf{d}}^{(J)}$ and $n$ is the number of data points. A singular value decomposition is normally performed on $M=U\Sigma V^{T}$, and the $i$th principal component for data point $J$ is now given by $\Sigma_{ii}U_{Ji}$. This works for discrete data, but in the case of continuous data (such as likelihood functions) we must take a slightly different approach.

We construct a matrix of dot products,

Since this matrix can also be expressed as $MM^{T}=U\Sigma V^{T}V\Sigma U^{T}=U\Sigma^{2}U^{T}$, we can find the principal components of our data by finding the eigenvalues and eigenvectors of the matrix of dot products. If we consider the case where data points ${\mathbf{d}}^{(J)}$ are the square roots of the probability distribution predicted from parameter combination ${\mbox{\boldmath$\theta$}}_{J}$ then, using the dot product for our $N$ replicated system, we can write out the components of $(MM^{T})$ as:

In the limit where the number of replicas go to zero, the matrix of dot products becomes:

In the case where the number of replicas is not a whole number, and in the limit where it tends to zero, the matrix of dot products is no longer positive definite. As a result, we can obtain positive and negative eigenvalues from its decomposition, leading to real and imaginary principal components.

In order to include experimental data into this plot, a probability distribution from the data must be generated. The dot product between these measurements and predicted distributions can then be calculated, and the results used in the matrix of dot products in Eq. S19 to find it’s projection along the principal components. The distance from the point to the manifold is given by our intensive distance from the point to the best fit.