SODA: Detecting Covid-19 in Chest X-rays with Semi-supervised Open Set Domain Adaptation

We summarize the symbols used in the paper and their descriptions in Table 1.

An overview of the proposed Semi-supervised Open Set Domain Adversarial network (SODA) is shown in Fig. 1. Given an input image $\mathbf{x}$, it will be first fed into a feature extractor $G_{f}$, which is a Convolutional Neural Network (CNN), to obtain its hidden feature $\mathbf{h}$ (green part). The binary classifier $G_{y_{l}}$ (part of the multi-label classifier $G_{y}$) takes $\mathbf{h}$ as input, and will predict the probability $\hat{y}_{l}$ for the label $l\in\mathcal{L}$ (blue part).

We propose a novel two-level alignment strategy for extracting the domain invariant features across the source and target domain. On the one hand, we perform domain alignment (Section 3.2), which leverages a general domain discriminator $D_{g}$ to minimize the domain-level feature discrepancy. On the other hand, we emphasize the alignment of common labels $\mathcal{L}^{c}$ (Section 3.3) by introducing another domain discriminator $D_{c}$ for images associated with common labels. For labeled images in $\mathcal{D}^{s}$ and $\mathcal{D}^{t^{\prime}}$, we compute loss for $D_{c}$ and conduct back-propagation during training only if the input image $\mathbf{x}$ is associated with a common label $l\in\mathcal{L}^{c}$. As for unlabeled data in $\mathcal{D}^{t}$, we propose a common label recognizer $R$ to predict the probability $\hat{r}$ that an image $\mathbf{x}$ has a common label, and use $\hat{r}$ as a weight in the losses of $D_{c}$ and $D_{g}$.

Domain adversarial training (Ganin et al., 2016) is the most popular method for helping feature extractor $G_{f}$ learn domain-invariant features such that the model trained on the source domain can be easily applied to the target domain. The objective function of the domain discriminator $D_{g}$ can be written as:

where $\hat{{d}}_{g}$ denotes the predicted probability that the input image belongs to the source domain.

In SODA, we use a Multi-Layer Perceptron (MLP) as the general domain discriminator $D_{g}$.

In the field of adversarial domain adaptation, most of the existing methods only leverage a general domain discriminator $D_{g}$ to minimize the discrepancy between the source and target domain. Such a practice ignores the label structure across domains, which will result in false alignment and even negative transfer (Pei et al., 2018; Wang et al., 2019). If we only use a general domain discriminator $D_{g}$ in the open set domain adaptation setting (Definition 2.1 and Definition 2.2), it is possible that the feature extractor $G_{f}$ will map the target domain images with a common label $l\in\mathcal{L}^{c}$, “Pneumonia”, and the source domain images with a specific label $l\in\bar{\mathcal{L}}^{s}$, e.g., “Cardiomegaly”, to similar positions in the hidden space, which might lead to the classifier miss-classifying a “Pneumonia” image in the target domain as “Cardiomegaly”.

To address the problem of the miss-matching between the common and specific label sets, we propose a domain discriminator $D_{c}$ to distinguish the domains for the images with a common label. For the labeled data from the source domain $\mathcal{D}^{s}$ and the target domain $\mathcal{D}^{t}$, we know whether an image $\mathbf{x}$ has a common label or not, and we only calculate the loss $L_{d_{c}}$ for $D_{c}$ on the samples with common labels:

where $\hat{d}_{c}$ denotes the predicted probability that the input images is associated with a common label.

However, a large number of images in the target domain are unlabeled, and thus extra effort is required for determining whether an unlabeled image is associated with a common label. To address this problem, we propose a novel common label recognizer $R$ to predict the probability $\hat{r}$ whether an unlabeled image has at least one common label. The probability $\hat{r}$ will be used as a weight in the loss function of $D_{c}$⁵⁵5Note that gradients stop at $\hat{r}$ in the training period.:

In addition, we also use $\hat{r}$ to re-weigh unlabeled samples in $D_{g}$ (Equation 1) to further emphasize the alignment of common labels:

Finally, the recognizer $R$ is trained on the labeled set $\mathcal{D}^{s}\cup\mathcal{L}^{t^{\prime}}$ via cross-entropy loss:

The overall objective function of SODA can be written as a min-max game between classifiers $G_{y}$, $R$ and discriminators $D_{g}$, $D_{c}$:

To investigate the effects of domain adaptation and demonstrate the performance improvement of the proposed SODA, we present the average AUC-ROC scores for all models in Table 2. Comparing the results for ResNet50 and DenseNet121, we observe that deeper and more complex models achieve better classification performance. For the effects of domain adaptation, it is obvious that the domain adaptation methods (DANN, PADA, and SODA) outperform those fine-tuning based transfer learning methods (ResNet50 and DenseNet121). Furthermore, the proposed SODA achieves higher AUC scores on both COVID-19 and Pneumonia than DANN and PADA, demonstrating the effectiveness of the proposed two-level alignment.

Proxy $\mathcal{A}$-Distance (Ben-David et al., 2007) has been widely used in domain adaptation for measuring the feature distribution discrepancy between the source and target domains. PAD is defined by

where $\epsilon$ is the domain classification error (e.g. mean absolute error) of a classifier (e.g. linear SVM (Cortes and Vapnik, 1995)).

Following (Ganin et al., 2016), we train SVM models with different $C$ and use the minimum error to calculate PAD. In general, a lower $d_{\mathcal{A}}$ means a better ability for extracting domain invariant features. As shown in Fig. 2, SODA has a lower PAD compared with the baseline methods, which indicates the effectiveness of the proposed two-level alignment strategy.

We use t-SNE to project the high dimensional hidden features $\mathbf{h}$ extracted by DANN, PADA, and SODA to low dimensional space. The 2-dimensional visualization of the features in the target domain is presented in Fig. 3, where the red data points are image features of “Pneumonia” and the blue data points are image features of “COVID-19”. It can be observed from Fig. 3 that SODA performs the best for separating “COVID-19” from “Pneumonia”, which demonstrates the effectiveness of the proposed common label recognizer $R$ as well as the domain discriminator for common labels $D_{c}$.

Grad-CAM (Selvaraju et al., 2017) is used to visualize the features extracted from all compared models. Fig. 4 shows the Grad-CAM results on seven different COVID-19 positive chest x-rays. These seven images have annotations (small arrows and box) indicating the pathology locations. We observe that ResNet50 and DenseNet121 can focus wrongly on irrelevant locations like the dark corners and edges. In contrast, domain adaptation models have better localization in general, and our SODA model gives more focused and accurate pathological locations than other models compared. In addition, we consult a professional radiologist with over 15 years of clinical experience from Wuxi People’s Hospital and received positive feedback on the pathological locations as indicated by the Grad-CAM of SODA. In the future, we plan to do a more rigorous evaluation study with more inputs from radiologists. We believe the features extracted from SODA can assist radiologists to pinpoint the suspect COVID-19 pathological locations faster and more accurately.

Domain adaptation is an important application of transfer learning that attempts to generalize the models from source domains to unseen target domains (Ganin and Lempitsky, 2015; Ganin et al., 2016; Tzeng et al., 2017; Tzeng et al., 2014; You et al., 2019; Jing et al., 2018; Wang et al., 2020). Deep domain adaptation approaches are usually implemented through discrepancy minimization (Tzeng et al., 2014) or adversarial training (Ganin and Lempitsky, 2015; Ganin et al., 2016; Tzeng et al., 2017). Adversarial training, inspired by the success of generative adversarial modeling (Goodfellow et al., 2014), has been widely applied for promoting the learning of transfer features in image classification. It takes advantage of a domain discriminator to classify whether an image is from the source or target domains. On top of these methods, a couple of works have been presented for exploring the high-level structure in the label space, which aim at further improving the domain adaptation performance for multi-class image classification  (Wang et al., 2019) or fundamentally solving the application problem when the label sets from source domains and target domains are different  (You et al., 2019). In order to meet the latter target, more and more researchers have started to study the open set domain adaptation problem, in which case the target domain has images that do not come from the classes in the source domain  (You et al., 2019; Panareda Busto and Gall, 2017). Universal domain adaptation is the latest method that is proposed through using an adversarial domain discriminator and a non-adversarial domain discriminator to successfully solve this problem. (You et al., 2019). Although domain adaptation has been well explored, its application in medical imaging analysis, such as domain adaptation for chest x-ray images, is still under-explored.

Semi-supervised learning is a very important task for image classification, which can make use of both labeled and unlabeled data at the same time (Saito et al., 2019). Recently it has been used to solve image classification problems on a very large (1 billion) set of unlabelled images (Yalniz et al., 2019). In spite of many progresses that have been made with unsupervised domain adaptation methods, the domain adaptation with semi-supervised learning has not yet been fully explored.

There has been substantial progress in constructing publicly available databases for chest x-ray images as well as a related line of works to identify lung diseases using these images. The largest public datasets of chest x-ray images are Chexpert (Irvin et al., 2019) and ChestXray14 (Wang et al., 2017), which respectively include more than 200,000 and 100,000 chest x-ray images collected by Stanford University and National Institute of Healthcare. The creation of these datasets have also motivated and promoted the multi-label chest x-ray classification for helping the screening and diagnosis of various lung diseases. The problems of disease detection (Wang et al., 2017; Irvin et al., 2019; Wang et al., 2018) and report generation using chest x-rays (Jing et al., 2017; Li et al., 2018; Jing et al., 2019; Biswal et al., 2020) are fully investigated and have achieved much-improved results upon recently. However, there have been very few attempts for studying the domain adaptation problems with the multi-label image classification problem using chest x-rays.