Hypothesis-Driven Deep Learning for Out of Distribution Detection

There is an increasing trend of using black-box machine learning (ML) for high-stakes prediction problems that deeply affect human lives, such as healthcare. Deep learning models inherently provide no explanation of their predictions in a human-interpretable manner. Deploying such models despite the lack of transparency and accountability can have (and already has) severe consequences [10]. Such problems are worsened when models are only trained on a small subset of the target population. For instance, when training black-box models to diagnose bacterial infections, it is infeasible to collect data spanning the entire population of bacteria species, so only a small subset of bacterial species is used instead [3, 4]. Formally, given a population of $N$ bacteria species, the common approach is to sample $n$ species ($n\ll N$), train a classifier within that subset, and report their performance on unseen samples from the $n$ species themselves. Such classifiers, if deployed in the real-world, are bound to receive inputs that appear similar to the training data, but are outside the $n$ species. Thus, a classifier untested on samples beyond the $n$ species may perform unpredictably on real-world data, which makes them unsafe for clinical use. To mitigate such problems, we require trustworthy models that not only classifies samples from known categories well, but also detects when samples cannot be assigned to those categories [2]. For instance, a trustworthy digit classifier should not only classify digit inputs correctly, but allow to detect non-digit inputs and reject them instead of blindly classifying them into existing categories.

Hypothesis testing is a fundamental concept in statistics that allows us to draw conclusions about an entire population based on a representative sample. At its core, hypothesis testing involves making assumptions about population parameters and then rigorously assessing whether the sample data provides enough evidence to support or reject these assumptions. In this paper, we reformulate the OoD detection problem as a hypothesis test, where we check for differences in OoD metrics between InD and OoD groups through a resampling approximation of the null distribution. The OoD metrics are chosen depending on a model’s inductive biases, and our test is independent of this choice. Further, we propose a technique based on leave-out training to quantify a model’s discriminating power for near-OoD inputs (i.e., same domain, different class). We test our method on several model architectures, datasets, and label splits, and show that it quantifies differences between InD and OoD data in an interpretable way. Overall, our method provides a basis for making informed decisions from classifiers trained on a subset of labels.

Let $D_{T}=\{x_{i},y_{i}\}_{i=1}^{N_{T}}$ be training data, $D_{V}=\{x_{i},y_{i}\}_{i=1}^{N_{V}}$ be validation data, $D_{I}=\{x_{i},y_{i}\}_{i=1}^{N_{I}}$ be InD test data, and $D_{O}=\{x_{i},y_{i}\}_{i=1}^{N_{O}}$ be OoD test data. Here, $x\in\mathbb{R}^{D}$ and $y\in\omega$. Our goal is to quantify how a model $\phi$ that was trained on $D_{T}$ and validated on $D_{V}$ would transform $D_{I}$ and $D_{O}$, using an ensemble of OoD metrics $M(x_{i};\phi):\mathbb{R}^{D}\rightarrow\mathbb{R}^{L}$ where $M=\{M_{1},\ldots,M_{L}\}$. The $L$ metrics are computed at multiple hidden layer activations of a model. We leave the choice of OoD metrics $M$ for future investigation, and instead, rely on established OoD metrics in the literature. We then define the following null hypothesis: H0: The new sample belongs to one of $\omega$ groups Next, using $P$ random samples of valid assignments $\pi$, we compute the MRPP statistic $\delta(\pi)$ for each assignment of $D_{I}$ and $D_{O}$ to $\omega$. Finally, we compute the proportion (p-value) of $\pi$ that yield $\delta(\pi)\leq\delta(\pi_{obs})$, and use it to quantify the difference between the model’s latent responses in an interpretable way.

Here, we use a labeled dataset, AMRB [1], having single-cell bacteria images taken from 21 strains across 5 species $\{Ab,Bs,Ec,Kp.Sa\}$. From this dataset, we take 2 subsets based on species, $A=\{Ab,Ec,Kp\}$ and $B=\{Bs,Sa\}$. Similarly, we take subsets from MNIST and CIFAR10 based on class label, $A=\{0,1,2,3,4\}$ and $B=\{5,6,7,8,9\}$ for model validation. Next, we train separate classifiers for every dataset and subset, and report their observed test accuracy (see Table 1). We also train separate auto-encoders, ResNet-AE, to learn a label-free encoding of input data. All models were implemented and trained using PyTorch Lightning on an NVIDIA A100 GPU.

For the ResNet-50 and ResNet-18 models, we use the standard, ImageNet pre-trained implementation provided in PyTorch. For the ResNet-AE model, we implemented a convolutional encoder $F$ and decoder $G$ with residual connections, and a classifier $C$ on the bottleneck of the auto-encoder. The ResNet-50 and ResNet-18 models were trained using cross-entropy loss. The ResNet-AE model was trained using reconstruction (MSE) loss, while the ResNet-CAE model was trained using both cross-entropy and MSE losses.

In our experiments, we use $\lambda=0.5$. The model parameters were optimized for 100 epochs, using Adam with a learning rate of $0.001$. Upon training, we observe that ResNet-50 models yield a high accuracy across all cases. To cover a range of loss functions and model architectures, we choose the ResNet-50, ResNet-CAE, and ResNet-AE models to validate our method.

In this paper, we present an approach to quantify whether a sample is out-of-distribution to a deep neural network. We formulate our method as a two-sample hypothesis test performed on an ensemble of OoD metrics. In particular, we use the Multi-Response Permutation Procedure (MRPP) statistic to quantify the dissimilarity of OoD metrics across two groups, and then recompute this statistic across random permutations of group assignments to determine the significance of the true observation. The null distribution obtained in this manner, provides an interpretable basis for decision-making. We validate our method on a toy problem created using MNIST and CIFAR10 datasets, and a domain problem of detecting an unseen bacteria species for a trained classifier.

The proposed method can be used on any trained model, given that OoD metrics can be extracted from it. Yet, some OoD metrics are limited to certain model architectures, or require access to a set of anchor points for their computation (e.g. KNN metrics). Having a validation dataset would provide a source of anchor points to compute such OoD metrics. While we use the MRPP statistic in our method, one could use any function of form $\mathbb{R}^{d}\rightarrow\mathbb{R}$ (e.g., Linear Regression + AUC) as the test statistic. However, the form of group difference measured would change with the statistic being used. Moreover, for the datasets we used, a mini-batch of $100$ samples and $3000$ permutations were sufficient to observe a stable behavior. However, the optimal sample size and permutations may vary depending on the data and domain.