A Model-based Semi-supervised Clustering Methodology

Clustering is the art of partitioning data into distinct and scientifically relevant classes by assigning labels to observations. Clustering is typically performed in an unsupervised setting, where none of the observed data initially have labels. In semi-supervised learning, some of the data have labels, but others do not; the goal is to classify the unlabeled data by assigning them to the same classes as the labeled data. In the gap between these two settings, there is semi-supervised clustering, in which some of the data have labels, and there may not be labeled data from every class. Furthermore, the total number of classes may be unknown. Just as in semi-supervised learning, leveraging the known labels allows for a more informed clustering.

There are many strategies for solving clustering problems. Some clustering procedures are based on heuristics that lack a principled justification, and many of the basic questions of clustering (e.g., how many classes there are) are often left to the intuition of the practitioner. Fraley and Raftery [9] review model-based clustering, which recasts the task of clustering as a model selection problem, which is well-studied in the field of statistics. The main assumption in model-based clustering is that the data are drawn from one of $G$ distributions, where each distribution represents a cluster. Model selection can be accomplished by computing approximate Bayes factors for competing models with different numbers and/or types of components.

Much of the recent work in model-based clustering has centered on variable selection. Raftery and Dean [18] proposed a greedy algorithm for model-based clustering in the unsupervised setting; it used the BIC to choose the number of clusters and the features to consider while clustering by proposing linear relationships between variables that influence clustering and those independent of clustering. Murphy et al. [15] adapted Raftery and Dean’s algorithm to semi-supervised learning. Maugis et al. [13, 12] consider many more scenarios of dependency structures between the clustering variables and the remaining variables than in [18]. Taking a different approach, Witten et al. [22] offer a framework for variable selection in clustering using sparse k-means or sparse hierarchical algorithms by modifying the corresponding objective function to include penalty constraints. In their simulations, they outperform the methods of [18] considerably.

The BIC is generally denoted as

$$BIC=2\mathcal{L}-d\log(n),$$

where $\mathcal{L}$ is the maximum of the likelihood, $d$ is the number of parameters estimated, and $n$ is the number of data used to estimate those parameters. Alternative information criteria generally differ by having different values for the penalty term. For example, the sample size adjusted BIC [5] is defined as

$$sBIC=2\mathcal{L}-d\log(\frac{n+2}{24}).$$

Because $BIC-sBIC=o(1),$ one may wonder of the efficacy of such a deviation from the BIC. This question is particularly salient when considering that the standard derivation of the BIC as the approximation to the integrated loglikelihood is only $O(1)$ accurate asymptotically (so that o(1) terms are negligible). In this paper, we will present such a modified BIC that is asymptotically equivalent to the standard BIC and thoroughly explore an analytical example of how such a modification could be useful.

In this paper, we quickly review model based clustering in Section 2. We state our derivation of a modified BIC to the semi-supervised case in Section 3 that is asymptotically equivalent to the standard BIC. We follow this with a detailed discussion of $o(1)$ equivalent information criteria, analytical calculations for a toy example, and simulations for a semi-supervised clustering example. Next, we apply our methods to the fly larvae behavioral dataset in Section LABEL:fly. Finally, we conclude the paper with a discussion of the results and extensions to our method.

Assume that the data $X_{1},X_{2},\dots,X_{n}$ are distributed i.i.d. according to a mixture model,

$$f_{\theta}(\cdot)=\sum_{k=1}^{G}\pi_{k}f_{\theta_{k}}(\cdot),$$

where $\theta_{k}$ are the parameters of the $k^{\text{th}}$ component of the mixture and $G$ is the total number of components. It can be shown that this is equivalent to the following generative process: for each $i\in\{1,2,\dots,n\}$

1. (1)

   draw $Z_{i}$ from $\text{multinomial}\left(\pi_{1},\pi_{2},\dots,\pi_{G}\right)$

2. (2)

   draw $X_{i}$ from $f_{\theta_{Z_{i}}}$.

If we consider each of the $G$ components as representing a single cluster, then the problem of clustering the data is that of unveiling each latent $Z_{i}$ from the generative process. The expectation-maximization (EM) algorithm can be used to find the maximum likelihood estimates for the parameters of the model and the posterior probabilities of cluster membership for each $X_{i}$ (cf. [8]).

With this formulation, we have a fully probabilistic clustering scheme. We can set the cluster of the $i^{\text{th}}$ observation to the maximum a posteriori estimate:

$$\hat{Z_{i}}=\text{argmax}_{k}\pi_{k}f_{k}(x_{i}).$$

Note that the posterior probability can give us a measure of confidence about the $i^{\text{th}}$ classification.

Here, we have used $f_{\theta_{k}}$ as a general density function, but it should be noted that the multivariate Gaussian distribution is the most common choice of distribution. We will eventually use an information criterion, such as the Bayesian Information Criterion (BIC), to assess the relative quality of different clusterings. Then, the number of clusters can be chosen using the BIC, which rewards model fit and penalizes model complexity.

By considering different constraints to the covariance matrices in the Gaussian components, we can reduce the number of parameters estimated, thus lowering the influence of the penalty term in the BIC. This allows for simpler clusters to be chosen over complex clusters. Celeux and Govaert [6] consider a spectral decomposition of the $k^{\text{th}}$ component’s covariance matrix,

$$\Sigma_{k}=\lambda_{k}D^{T}_{k}A_{k}D_{k},$$

where $\lambda_{k}$ represents the largest eigenvalue, $A_{k}$ is a diagonal matrix whose largest entry is $1$, and $D_{k}$ is a matrix of the corresponding eigenvectors. The interpretation of each term as it relates to cluster $k$ is as follows: $\lambda_{k}$ represents the volume, $D_{k}$ the orientation, and $A_{k}$ the shape. By forcing one or more of these terms to be the same across all clusters, we can reduce the number of parameters to be estimated from $Gd+G\frac{d(d+1)}{2}$ in the unconstrained case ($\Sigma_{k}=\lambda_{k}D^{T}_{k}A_{k}D_{k}$) to $Gd+G-1+\frac{d(d+1)}{2}$ in the most constrained case ($\Sigma_{k}=\lambda D^{T}AD$). We can also force additional constraints, such $D_{k}=I$ and/or $A_{k}=I$, leading to the simplest model: $\Sigma_{k}=\lambda I$ with only $G(d+1)$ parameters to estimate.

The main theoretical contribution of this paper is to derive an adjustment to the BIC for the semi-supervised clustering setting followed by an exploration about $o(1)$ equivalent information criteria. We will set forth a formalization of a more general setting, derive the BIC in this broader setting, and then apply our result to the special case of semi-supervised clustering. The modified BIC then represents a principled measure by which to choose the number of clusters and the variables for clustering when performing semi-supervised clustering.

Consider $n=\sum_{i=1}^{C}n_{i}$ independent random variables

	$$\displaystyle X_{1}^{(1)},X_{2}^{(1)},\dots,X_{n_{1}}^{(1)}\overset{\text{i.i.d.}}{\sim}f_{\theta_{1}},$$
	$$\displaystyle X_{1}^{(2)},X_{2}^{(2)},\dots,X_{n_{2}}^{(2)}\overset{\text{i.i.d.}}{\sim}f_{\theta_{2}},$$
	$$\displaystyle\vdots$$
	$$\displaystyle X_{1}^{(C)},X_{2}^{(C)},\dots,X_{n_{C}}^{(C)}\overset{\text{i.i.d.}}{\sim}f_{\theta_{C}},$$

where $\theta_{1}\in\Theta_{1}\subset\mathbb{R}^{d_{1}}$ and $\theta_{j}\in\Theta_{j}\subset\Theta_{1}$ for $j=2,3,\dots C$ and

$$(f_{\theta_{1}},f_{\theta_{2}},\dots,f_{\theta_{C}})\in M=\{(f_{\theta^{(1)}},f_{\theta^{(2)}},\dots,f_{\theta^{(C)}}):\theta=(\theta^{(1)},\dots,\theta^{(C)})\in\Theta_{1}\times\Theta_{2}\times\dots\times\Theta_{C}\}.$$

Collect all of the $X^{\prime}s$ into set $D$.

Model $M$, while more general than strictly necessary, encompasses the semi-supervised case. Consider the first group of $n_{1}$ random variables as those for which we do not have labels and each of the other $C-1$ groups as those whose labels we know. Then, for a proposed total number of clusters $G\geq C-1$, we assume

$$f_{\theta_{1}}(x)=\sum_{j=1}^{G}\pi_{j}\phi(x;\mu_{j},\Sigma_{j}),$$

where $\phi(\cdot;\mu_{j},\Sigma_{j})$ is a multivariate normal pdf with mean $\mu_{j}$ and covariance matrix $\Sigma_{j}$, and $\sum_{j=1}^{G}\pi_{j}=1$. Also, for each $k\in\{2,3,\dots,C\}$,

$$f_{\theta_{k}}(x)=\phi(x;\mu_{j_{k}},\Sigma_{j_{k}}),$$

where the double subscript $j_{k}$ is to account for possible relabeling. It follows that

$$\theta_{1}=\left((\pi_{1},\pi_{2},\dots,\pi_{G}),(\mu_{1},\mu_{2},\dots,\mu_{G}),(\Sigma_{1},\Sigma_{2},\dots,\Sigma_{G})\right)\in\Theta_{1},$$

where

$\displaystyle\Theta_{1}=\{((\pi_{1},\pi_{2},\dots,\pi_{G}):\sum_{i=1}^{G}\pi_{i}=1,\pi_{i}\in[0,1]\}\times\mathbb{R}^{d\times G}\times\{(\Sigma_{1},\Sigma_{2},\dots,\Sigma_{G})|\Sigma_{i}\succeq 0,\Sigma_{i}\in\mathbb{R}^{d\times d}\}.$

Then, for each $k\in\{2,3,\dots,C\}$, $\Theta_{k}$ is the same as $\Theta_{1}$ except that the mixing coefficients $(\pi_{1},\pi_{2},\dots,\pi_{G})$ are constrained such that $\pi_{j_{k}}=1$ and all other $\pi_{i}=0$.

With this model, we can derive the Expectation step (E-step) and Maximization (M-step) of the EM algorithm. Note that semi-supervised EM is not new; indeed, similar calculations can be found in [14] Section 2.19 and [19].

Here, we generalize the BIC to the semi-supervised case with the aim of not overly penalizing for supervised data. The modified BIC approximation to the integrated likelihood of model $M$ is

$$2\log(P(D|M))\approx 2\log(P(D|\hat{\theta},M))-d\log(n_{1})=BIC^{*}_{M},$$

where $\hat{\theta}$ is the maximum likelihood estimate (MLE).

We provide the derivation in the appendix. Compare the above result to the classical BIC approximation:

$$BIC\approx 2\log(P(D|M))\approx 2\log(P(D|\hat{\theta},M))-d\log(n).$$

The derivation of the BIC (adjusted and normal) involves an O(1) term, which is dropped at the end. Due to this, the difference between the two BICs is a smaller penalty in the semi-supervised case (i.e. $d\log(n_{1})$ vs. $d\log(n)$, which is a $o(1)$ difference). This can be interpreted as not penalizing more complicated cluster structures for the supervised data. We will discuss the merit of such a deviation in Section 4.

It is known that finite mixture models do not meet the regularity conditions for the BIC approximation used to approximate the integrated likelihood. However, [9] comment that it is still an appropriate and effective approximation in the paradigm of model-based clustering. Thus, we will not be too remiss in considering Theorem LABEL:BIC as applicable when performing semi-supervised clustering. Because the model $M$ is appropriate for the semi-supervised case, we can use the modified BIC as a way to calculate Bayes factors for different number of total clusters $G$, subsets of variables to be included in the clustering, and parameterizations of $\Sigma_{i}$. The largest BIC value will therefore correspond to our chosen model for clustering. We will call the semi-supervised clustering algorithm using the BIC from Theorem LABEL:BIC to make the aforementioned selections ssClust henceforth.

Here, we comment on the use of different penalty terms. Suppose that we define an adjusted BIC, say $BIC^{\prime}$ as a function of $m\in[1,\inf)$ as $BIC^{\prime}(m)=2\mathcal{L}-d\log(m)=BIC-d\log(\frac{m}{n}).$

When does using such an adjustment matter? Consider two models, say $M_{0}$ and $M_{1}$, in competition for selection. Fix an $m$. Suppose that

1. (i)

   $d_{1}>d_{0}$

2. (ii)

   $BIC_{0}>BIC_{1}$ and

3. (iii)

   $BIC^{\prime}_{0}(m)<BIC^{\prime}_{1}(m).$

In this case, we see that under the original definition of the BIC, we would choose $M_{1}$ over $M_{0}$, and with the modified definition, we would do the opposite.

In order to analyze this under some assumptions, define two statistical tests:

$$T(\omega)=\mathbbm{1}\{BIC_{1}>BIC_{0}\}$$

and

$$T^{\prime}_{m}(\omega)=\mathbbm{1}\{BIC^{\prime}_{1}(m)>BIC^{\prime}_{0}(m)\}.$$

When do $T$ and $T^{\prime}$ differ?

1. Case 1:

   $T=1$ and $T^{\prime}_{m}=0.$ For $m\leq n$ this is impossible.

2. Case 2:

   $T=0$ and $T^{\prime}_{m}=1.$ We have two subcases.

   1. (a)

      Suppose $M_{1}$ is the true model. We would be correct in choosing $T^{\prime}_{m}$ over $T$.

   2. (b)

      Suppose $M_{0}$ is the true model. Note that in this case, we would make a mistake by choosing $T^{\prime}_{m}$ over $T$.

We would like to analyze the probabilities in case 2. Preferably, (2.a) is much more probable than (2.b). Unfortunately, under (2.a) the distribution of $W_{1,0}$ is only known for certain cases. For example, with local alternatives of the form $\theta_{n}=\theta_{0}+\frac{\Delta}{\sqrt{n}},$ $W_{1,0}(d_{1}-d_{0})$ is known to have noncentral chi square distribution with $df=d_{1}-d_{0}$ and the appropriate non-centrality parameter. Since this result requires rather quixotic assumptions to hold, we will not theoretically bound the probabilities for more general cases; instead, we will defer our analysis to a specific example.

We can say more about (2.b) with some assumptions.

In one of the motivating applications for this work, classes of neurons in Drosopholia larvae are controlled using optogenetics (cf. [1] regarding optogenetics). In [20], they observe the reactions of the affected larvae to stimuli in high-throughput behavioral assays. The goal is to determine which classes of neurons cause similar changes in behavior when deactivated.

We initially collected data on $n=37780$ larvae grouped into $b=2062$ dishes. By changing the optogenetic procedure, $\ell=11$ known lines are created, pbd1, 38a1, 61d0, ppk1, 11f0, pbd2, 38a2, pbd3, ppk2, iav1, and 20c0. Of these lines, we discarded the larvae in pbd3 and ppk2 because they had less than 40 larvae each. Further, we discarded all larvae with an unknown line. After curating the data, we now have $n=7730$ larvae. Each larva is observed while responding to various stimuli. Vogelstein et al. citevogelstein2014discovery expand on the methods of [16] and [17] and describe how the observations are embedded into $\mathbb{R}^{d}$, where here $d=30$. We use the method presented in [24] to select the elbow of the scree plot to further reduce the data to $d=14$ dimensions. We did not perform any additional feature selection.

For each Monte Carlo replicate, we use a small subset of the data where the line was known (101 randomly chosen animals from each of the 9 remaining lines) along with $m=0,1,2,\dots 8$ pre-labeled data randomly chosen from the 101 animals in each line. We wrote an R package entitled ssClust to perform semi-supervised GMM with similar options to the popular Mclust software, which is an R package for GMM [10]. We cluster the points using both ssClust and Mclust. Then, we compute the ARI against the line type for both methods.

We observe that the initialization strategy of using ss-k-means++ instead of hierarchical clustering results in a significant improvement to the ARI even with no supervision. We find that a single animal per line significantly improves the clustering results, as expected (cf. Figure 6). Further, we can see that there are diminishing returns on additional supervision starting at $3$ supervised examples per line.

Previously explored approaches to semi-supervised clustering include a modified K-means, which can be seen as our algorithm constrained to spherical and identical covariance matrices without the adjusted BIC for model selection (cf. [2]). Others have used latent variables representing clusters and cluster-dependent distributions to give a probabilistic approach to a slightly different problem, where instead of labels being known, only pairwise constraints in the form of must-link and cannot-link are given (cf. [3]). [21] applied an appropriately modified K-means to this problem. Finally, [7] and [23] offer a different approach to semi-supervised clustering involves training a measure to account for constraints or labels.

The main contribution of this paper over previous works is presenting a probabilistic framework for semi-supervised clustering and deriving an information criterion consistent with the framework to allow selection of number of clusters and/or clustering variables. With the corrected BIC, we found that in the simulated examples and fly larvae dataset our method outperformed the most comparable method, Mclust. In the fly dataset, ssClust was able to recover lines better and yielded a lower answering time more often for the question of whether two particular lines were different, behaviorally. This indicates that incorporating even a small amount of information can help guide the clustering process.

Future areas of research will involve handling the more nuanced must-link and cannot-link constraints, which are flexible enough to encompass the labels-known problem we have explored in this paper.

A Derivation of the BIC for the Semi-supervised Model
Assume the data are distributed according to a member of model $M$ described in Section 3.

Consider the integrated likelihood:

$$P(D)=\int P(D|\theta,M)P(\theta|M)d\theta,$$

(3)

where $P(\cdot)$ is a probability, pmf, or pdf where appropriate. Let

$$Q(\theta)=\log\left(P(D|\theta,M)P(\theta|M)\right),$$

the log of the posterior likelihood. Suppose that the posterior mode exists, say $\bar{\theta}$.

A second order Taylor expansion about $\bar{\theta}$ gives

$\displaystyle Q(\theta)$

$\displaystyle=$

$\displaystyle Q(\bar{\theta})+(\theta-\bar{\theta})^{T}\nabla Q(\bar{\theta})+\frac{1}{2}(\theta-\bar{\theta})^{T}\nabla^{2}Q(\bar{\theta})(\theta-\bar{\theta})+o(\|(\theta-\bar{\theta})\|_{2}^{2}).$

By the first order optimality necessary conditions, we know $\nabla Q(\bar{\theta})=0.$ Note $o(\|(\theta-\bar{\theta})\|_{2}^{2})\rightarrow 0$ faster than $\|(\theta-\bar{\theta})\|_{2}^{2}$ as $\theta\rightarrow\bar{\theta}$. By ignoring the last term, we will approximate $Q(\theta)$ with the truncated Taylor expansion:

$\displaystyle Q(\theta)\approx Q(\bar{\theta})+\frac{1}{2}(\theta-\bar{\theta})^{T}\nabla^{2}Q(\bar{\theta})(\theta-\bar{\theta}).$

Recalling (3), we may approximate $P(D|M)$ using a saddle point approximation:

	$\displaystyle P(D|M)$	$\displaystyle=$	$\displaystyle\int\exp\left(\log(P(D|\theta,M)P(\theta|M))\right)d\theta$
		$\displaystyle=$	$\displaystyle\int\exp(Q(\theta))d\theta$
		$\displaystyle\approx$	$\displaystyle\int\exp\left(Q(\bar{\theta})+\frac{1}{2}(\theta-\bar{\theta})^{T}\nabla^{2}Q(\bar{\theta})(\theta-\bar{\theta})\right)d\theta$
		$\displaystyle=$	$\displaystyle\exp\left(Q(\bar{\theta})\right)\int\exp\left(\frac{1}{2}(\theta-\bar{\theta})^{T}\nabla^{2}Q(\bar{\theta})(\theta-\bar{\theta})\right)d\theta$
		$\displaystyle=$	$\displaystyle\exp\left(Q(\bar{\theta})\right)\int\exp\left(-\frac{1}{2}(\theta-\bar{\theta})^{T}(-\nabla^{2}Q(\bar{\theta}))(\theta-\bar{\theta})\right)d\theta.$

Recognize the integral as proportional to the density of a multivariate Guassian with mean $\bar{\theta}$ and covariance $-\nabla^{2}Q(\bar{\theta})$. Let $H=-\nabla^{2}Q(\bar{\theta}).$ Then, we have

$\displaystyle P(D|M)$

$\displaystyle\approx$

$\displaystyle\exp\left(Q(\bar{\theta})\right)(2\pi)^{\frac{d}{2}}\det(H^{-1}))^{\frac{1}{2}},$

(4)

where $d$ is number of free parameters in $\theta$. If the conditions in Proposition 3.4.3 in [4] hold, we have for $n$ sufficiently large, $H\approx I(\bar{\theta}),$ where $I(\bar{\theta})$ is the Fisher information matrix.

Taking $2\log(\cdot)$ of both sides, (4) becomes

	$\displaystyle 2\log(P(D|M))$	$\displaystyle\approx$	$\displaystyle 2Q(\bar{\theta})+d\log(2\pi)+\log(\det(I(\bar{\theta})^{-1}))$
		$\displaystyle=$	$\displaystyle 2\log(P(D|\bar{\theta},M))+2\log(P(\bar{\theta}|M))+d\log(2\pi)-\log(\det(I(\bar{\theta})).$

Now, we must calculate $-\log(\det(I(\bar{\theta}))$. Define $n\equiv\sum_{i=1}^{C}n_{i}.$ Observe

	$\displaystyle I(\theta)$	$\displaystyle=$	$\displaystyle\text{Var}\left[\frac{\partial}{\partial\theta}\log(p(X,\theta)\right]$
		$\displaystyle=$	$\displaystyle\text{Var}\left[\sum_{i=1}^{n}\frac{\partial}{\partial\theta}\log(p(X_{i},\theta))\right]$
		$\displaystyle=$	$\displaystyle\sum_{i=1}^{C}n_{i}I(\theta_{i})\text{ by independence.}$

Henceforth, we will use $I_{j}$ to denote $I(\theta_{j})$. We will assume that $I_{1}$ is positive definite, so that it can be written as

$$I_{1}=S_{1}^{T}S_{1}$$

for some non-singular matrix $S_{1}$. Let $J$ denote the $d$-dimensional identity matrix. For notational purposes, let

$$n^{*}:=\max_{2\leq j\leq C}(n_{j}).$$

Observe

	$\displaystyle\text{det}\left(\sum_{i=1}^{C}n_{i}I(\theta_{i})\right)$	$\displaystyle=$	$\displaystyle\text{det}\left(S_{1}^{T}(n_{1}J+\left(S_{1}^{T}\right)^{-1}\left(\sum_{i=2}^{C}n_{i}I(\theta_{i})\right)S_{1}^{-1})S_{1}\right)$
		$\displaystyle=$	$\displaystyle\text{det}(I_{1})\text{det}\left(n_{1}J+n^{*}\left(S_{1}^{T}\right)^{-1}\left(\sum_{i=2}^{C}\frac{n_{i}}{n^{*}}I(\theta_{i})\right)S_{1}^{-1}\right)$
		$\displaystyle=$	$\displaystyle\text{det}(I_{1})n_{1}^{d}\text{det}\left(J+\frac{n^{*}}{n_{1}}\left(S_{1}^{T}\right)^{-1}\left(\sum_{i=2}^{C}\frac{n_{i}}{n^{*}}I(\theta_{i})\right)S_{1}^{-1}\right)$
		$\displaystyle=$	$\displaystyle\text{det}(I_{1})n_{1}^{d}\text{det}\left(J+\frac{n^{*}}{n_{1}}B\right),$

where $B=\left(S_{1}^{T}\right)^{-1})(\sum_{i=2}^{C}\frac{n_{i}}{n^{*}}I(\theta_{i}))S_{1}^{-1}$. It should be noted that $B$ is a positive semi-definite matrix, as it is a $*-$transform of a sum of positive semi-definite matrices, so that Sylvester’s Theorem states that $B$ has the same inertia as a positive semi-definite matrix.

Next, we would like to describe the growth of $\text{det}\left(J+\frac{n^{*}}{n_{1}}B\right)$. For any eigenvalue of $J+\frac{n^{*}}{n_{1}}B$, say $\lambda$, we have by Weyl’s theorem

$$1\leq\lambda\leq 1+\frac{n^{*}}{n_{1}}\|B\|_{2}.$$

Observe

	$\displaystyle\|B\|_{2}$	$\displaystyle=$	$\displaystyle\max_{\{x\in\mathbb{R}^{d}:\|x\|_{2}=1\}}x^{T}Bx$
		$\displaystyle\leq$	$\displaystyle\max_{\{x\in\mathbb{R}^{d}_{1}:\|x\|_{2}=1\}}\|S_{1}^{-1}\|_{2}^{2}x^{T}\left(\sum_{j=2}^{C}\frac{n_{j}}{n^{*}}I_{j}\right)x$
		$\displaystyle\leq$	$\displaystyle\max_{\{x\in\mathbb{R}^{d}:\|x\|_{2}=1\}}\|S_{1}^{-1}\|_{2}^{2}\left(\sum_{j=2}^{C}\frac{n_{j}}{n^{*}}\|I_{j}\|_{2}^{2}\right)$
		$\displaystyle\leq$	$\displaystyle\max_{\{x\in\mathbb{R}^{d}:\|x\|_{2}=1\}}\|S_{1}^{-1}\|_{2}^{2}\left(\sum_{j=2}^{C}\|I_{j}\|_{2}^{2}\right),$

which is independent of $n$. Let $M_{2}\equiv\max_{\{x\in\mathbb{R}^{d}:\|x\|_{2}=1\}}\|S_{1}^{-1}\|_{2}^{2}\left(\sum_{j=2}^{C}\|I_{j}\|_{2}^{2}\right).$ Then, we have

	$\displaystyle\text{det}(J+\frac{n^{*}}{n_{1}}B)$	$\displaystyle=$	$\displaystyle\Pi_{m=1}^{d}\lambda_{m}\left(J+\frac{n^{*}}{n_{1}}B\right)$
		$\displaystyle\leq$	$\displaystyle\left(1+\frac{n^{*}}{n_{1}}(M_{2})\right)^{d}$
		$\displaystyle\leq$	$\displaystyle\left(1+\frac{n-n_{1}}{n_{1}}(M_{2})\right)^{d}$

The only term growing with $n$ above is $\frac{n-n_{1}}{n_{1}},$ the ratio of the supervised data over the unsupervised data. In general, it is usually much more expensive to obtain additional supervised data than unsupervised; thus, we find it reasonable to posit that $\frac{n-n_{1}}{n_{1}}\rightarrow 0$ in $n$ (i.e. is $o(1)$). In this case, $\log\left(\text{det}(J+\frac{n^{*}}{n_{1}}B)\right)$ is $o(1)$ by continuity and our bounds.

Hence,

	$\displaystyle\log\left(\text{det}\right(I(\theta)\left)\right)$	$\displaystyle=$	$\displaystyle\log\left(\text{det}(I_{1})\right)+d\log(n_{1})+\log\left(\text{det}(J+\frac{n^{*}}{n_{1}}B)\right)$		(5)
		$\displaystyle=$	$\displaystyle\log\left(\text{det}(I_{1})\right)+d\log(n_{1})+o(1).$		(6)

When the posterior mode is nearly or is equal to the MLE, as is the case when the prior on $\theta$ is uniform and $\Theta$ is finite (cf. Bickel and Doksum pp. 114), substitute $\hat{\theta}$, the MLE, for $\bar{\theta}$, the posterior mode. Then, we have

	$\displaystyle 2\log(P(D|M))$	$\displaystyle\approx$	$\displaystyle 2\log(P(D|\hat{\theta},M))+2\log(P(\hat{\theta}|M))+d\log(2\pi)-\log(\det(I_{1}(\bar{\theta}))$
		$\displaystyle=$	$\displaystyle 2\log(P(D|\hat{\theta},M))-d\log(n_{1})+o\left(1\right)+O(1)\text{ using equation }(2).$

Note that the terms of order less than $O(1)$ get washed out in the limit as $n\rightarrow\infty$. If we drop them, we have our derivation of the adjusted BIC. ∎