Active Image Synthesis for Efficient Labeling

Following the standard GAN [12, 35], we model the training set images as realizations of the distribution of the images of interest $\mathcal{X}:\mathcal{B}[\mathbb{X}]\mapsto[0,1]$. Here, $\mathbb{X}=\mathbb{R}^{n_{1}\times n_{2}}$ denotes the space of images with pixel size $n_{1}\times n_{2}$, and $\mathcal{B}[\mathbb{X}]$ is its Borel set [36]. Furthermore, in order to efficiently learn the generative mapping and later interpretation, we define a feature space $\mathbb{F}=[-1,1]^{r}$. Here, $r$ is the pre-defined dimension of the feature space, usually assumed to be much lower than that of the image space. We set a non-informative, uniform measure $\mathcal{U}$ on the feature space $\mathbb{F}$, which represents the lack of understanding of the feature space. The goal is to learn a generative mapping $G(\cdot):\mathbb{F}\mapsto\mathbb{X}$ which best pushforwards the uniform measure $\mathcal{X}^{\prime}=G_{\#}(\mathcal{U})$ and mimics the target measure $\mathcal{X}$. We use in this work the Wasserstein-1 metric [37] as the loss function:

where $\|\cdot\|_{2}$ is the $l_{2}$ norm, and the infimum is obtained with respect to all the possible joint distribution $\gamma:\mathcal{B}[\mathbb{X}\times\mathbb{X}]\mapsto[0,1]$ whose marginals are $\mathcal{X}$ and $\mathcal{X}^{\prime}$. We adopt the Kantorovich-Rubinstein dual form [37] of Wasserstein distance for efficient implementation:

Here $D(\cdot):\mathbb{X}\mapsto\mathbb{R}$ is an evaluating function and $\|D(\cdot)\|_{L}\leq 1$ represents that $D(\cdot)$ is Lipschitz-1 continuous [35].

We use a NN to approximate the generating mapping $G(\cdot)$, named generator, and another NN for the evaluating function $D(\cdot)$, named discriminator. The aim is to find the optimum of the following minimax function:

where $\mathcal{X}_{n}$ is the empirical measure for the training images with size $n$, and $\mathcal{U}$ is the uniform measure to be pushforwarded. Iterative training strategy can be adapted. Further discussion on the numerical implementation and the convergence analysis can be found in Section 3.3.

Assume for now the generating mapping $G(\cdot)$ is known. We are interested in finding an encoding mapping $E(\cdot):\mathbb{X}\mapsto\mathbb{F}$ to embed the images back to the feature space, which, to be shown in Theorem 1, is an inverse of $G(\cdot)$. Similar to the generating mapping, we use a NN to parametrize $E(\cdot)$, named encoder. Since the task here is to extract the feature vectors from images, a convolutional neural network (CNN) is used with mean square error (MSE) loss:

where $f\in\mathbb{F}$ is the corresponding feature vector associated with the image $\mathbf{X}$. The reason we use an MSE loss is due to the desired regression task here: we want a strong metric to ensure the sample-to-sample inverse of $G(\cdot)$. Furthermore, we want $E(\cdot)$ dedicated only on this inversion task, and therefore permits an efficient sampling method later in Section 4.1.

However, the difficulty is that the feature $f$ for the actual image $\mathbf{X}$ is unknown. In other words, $E(\cdot)$ cannot be learned from the dataset of actual images at hand. Therefore, we revise (4) to

where $\mathbf{X}^{\prime}=G(u)$ is the generated virtual images. Another advantage of using the virtual data points is that the data size of the virtual images can be large, since one may generate as many virtual images $\mathbf{X}^{\prime}$ as needed. We expect the encoder $E(\cdot)$ learned via (5) using virtual data points (instead of the actual images in training set) is still the inverse of the generator $G(\cdot)$. Formally, we have the following Theorem:

The reason for introducing the encoding mapping $E(\cdot)$ as the inverse of generating mapping $G(\cdot)$ is twofold. First, we can use $E(\cdot)$ to encode the actual images as vectors in the feature space $\mathbb{F}$. They can then be used as lighthouses in $\mathbb{F}$, and provide intuitive understanding of the feature space (we will discuss this later in Section 5.2.2). Second and perhaps more important, in the following sampling method, we want to sample virtual images for better predictive performance with a limited labeling budget. In our AISEL method (see Section 4.1), the sampling is performed in $\mathbb{F}$ rather than the image space $\mathbb{X}$, for its lower dimension and the physical meaning. Moreover, while sampling virtual images, we need guidance from the features of the actual images. For example, one may not want to sample images that are too similar to any of the actual images to better explore the whole $\mathbb{F}$. This can be achieved by introducing a separating distance between virtual images and actual images (we will come back to this in Section 4.1); this needs to encode the actual images to the feature space by $E(\cdot)$.

Putting everything together, the proposed GIN consists of three NNs: a generator $G(\cdot)$ for generating virtual images, an encoder $E(\cdot)$ for feature embedding, and a discriminator $D(\cdot)$ for computing the Wasserstein distance. Fig.1 illustrates the architecture of GIN. Note that in GIN, $G(\cdot)$ and $E(\cdot)$ is decoupled due to the limited training data. We present Algorithm 1 to train the proposed GIN. The first part of the algorithm is to train a generator $G(\cdot)$ parameterized by $\theta$, and the second part is to train an encoder $E(\cdot)$ parameterized by $\gamma$. The generator and discriminator are coupled trained as GAN, while the additional encoder is separately trained by the virtual images sampled by $G(\cdot)$. In the small-data situation, the proposed GIN along with the associated algorithm can achieve visual improvement in practice, compared to other methods like BiGAN; we will provide a detailed discussion in Section 5.1.2.

One may be interested in finding out how “real” the virtual images can be generated using the proposed Algorithm 1, since multiple heuristic strategies are involved (e.g., iterative training of $D(\cdot)$ and $G(\cdot)$, and clip). Furthermore, note that the above computation is done with samples of actual images, i.e., the empirical probability measure $\mathcal{X}_{n}$, instead of the original probability measure $\mathcal{X}$. Therefore, we have the following theorem for asymptotic convergence.

Theorem 2 suggests that, if we have enough training data, the generated images are real enough compared to the actual images. Specifically, it means the generated images and the measure $\mathcal{X}^{\prime}$ have the following two properties. First and most importantly, the supports of the two measures are the same, i.e., $supp(\mathcal{X}^{\prime})=supp(\mathcal{X})$, with probability 1.0. This is a natural corollary of Theorem 2. It means any generated virtual image $\mathbf{X}^{\prime}$ can be regarded as a draw from the measure of actual images $\mathcal{X}$, i.e., $p_{\mathcal{X}}(\mathbf{X}^{\prime})>0$, where $p_{\mathcal{X}}(\cdot)$ denotes the probability density of $\mathcal{X}$. In other words, the generated images are always physically meaningful. Moreover, besides their support, the two probability measures themselves are the same asymptotically. This means the probability of generating the same group of images (e.g., CT scans of male patients, or CT scans of patients with no complications) is the same, which is an implicit requirement when endowing the feature space with physical meaning and for the following sampling method. Though in practice we are dealing with a small-data situation, it is still appealing to have this asymptotic convergence property.

We start with using the entropy [38] to quantify the uncertainty of the native model $C(\cdot)$. For any input image $\mathbf{X}_{0}\in\mathbb{X}$ and the corresponding predicted label $y_{0}=C(\mathbf{X}_{0})$, we have

where, $y_{0}=[y_{0}[1],y_{0}[2],\cdots,y_{0}[K]]^{T}$. The reason for using entropy to quantify uncertainty can be explained as: (i) If we are sure about the class label of the input image, e.g., $y_{0}[1]=1$ and $y_{0}[k]=0,k=2,\cdots,K$; the corresponding entropy is zero, meaning no uncertainty exists. (ii) Consider another extreme situation that $y_{0}[k]=1/K,k=1,\cdots,K$. One can easily check this maximizes the entropy, reflecting the maximal uncertainty for the label of that image.

The image space $\mathbb{X}=\mathbb{R}^{n_{1}\times n_{2}}$ is too high dimensional to handle in reality. Since GIN is already obtained, we can measure the uncertainty (i.e., entropy), for any $f_{0}\in\mathbb{F}$ in the feature space:

Here, we select the features of the complementary dataset in the feature space $\mathbb{F}$, rather than in the high-dimensional image space $\mathbb{X}$. Besides the dimensionality, our $G(\cdot)$ can capture the intrinsic structure of the image space $\mathbb{X}$ – selecting features in $\mathbb{F}$ (and then generating images via $G(\cdot)$) can ensure the existence of physical meaning. This is because any generated images $G(f_{0})$ with any $f_{0}\in\mathbb{F}$ is physically meaningful thanks to the $G(\cdot)$, while randomly sampled $\mathbf{X}_{0}\in\mathbb{X}$ is most likely a matrix without any visual clue.

The entropy $h(\cdot):\mathbb{F}\mapsto[0,\log K]$ can also be viewed as a (unnormalized) probability density on the measurable space $\{\mathbb{F},\mathcal{B}[\mathbb{F}]\}$. We denote this uncertainty measure as $\mu_{h}$.

We then propose to select the best set of $m$ virtual images, by matching its empirical distribution to the uncertainty measure:

Here, $dist(\cdot,\cdot)$ is a distance metric, $f_{1:m}^{\prime}=\{f_{i}^{\prime}\}_{i=1}^{m}$ denote the selected features, and $\mathcal{F}^{\prime}_{m}$ denotes the empirical measure for $f_{1:m}^{\prime}$. Intuitively, (8) means to assign more points to higher uncertainty regions (of the native model), and therefore exploit those regions. Furthermore, if taking the Bayesian perspective, it can be viewed as changing the initial uniform distribution, i.e., the non-informative prior, to the posterior distribution of uncertainty given the actual training dataset.

Motivated by the literature in the statistical community [39, 40], we select the energy distance as the metric $dist(\cdot,\cdot)$ between distributions. Therefore, we minimize:

Note again, the above sampling optimization is conducted in the feature space. We observe from (9) that the selected features not only try to match the target uncertainty measure in the expectation sense (the first term), but also separate from one another (the second term). The separating property is of great importance; this is because any two selected features that are too close to each other can be viewed as a waste of the expensive labeling process.

Furthermore, the selected features should also be separated from the features of the actual images, again to avoid waste. This can be taken into account by the following modification of (9):

where $n$ is the size of the actual dataset and let $f^{\prime}_{i}=f_{i}$ for actual images with indies $i=m+1,\cdots,m+n$. Comparing (10) to (9), we notice the difference lies in the second term, where the separating property is incorporated not only between the selected features but also between the selected features and the features of actual images. We use (10) for sampling features, and then generate an AISEL dataset via $G(\cdot)$. The following theorem ensures the generated AISEL dataset follows the target uncertainty measure in distribution.

The proposed sampling strategy (10) reveals an important trade-off. Consider the first term, where the selected features are forced to be close to the target uncertainty measure. Since the density of our target measure (6) is high when the uncertainty is high, the selected features can be viewed as exploiting the highly uncertain regions. Now consider the second term, where the separating distance is maximized. This suggests the selected features should be away from (i) one another and (ii) the features for actual images. Therefore, selected features are forced to be spread out and fill the whole feature space – they explore the entire feature space. Putting both parts together, selected features for virtual images jointly exploit the highly uncertain regions and explore the entire image space. This trade-off of our AISEL dataset will be shown as the key to improve the classification performance.

We want to make a few remarks here. First, the proposed sampling method, specifically the uncertainty measure (7), is specifically for the classification problem at hand. With a different uncertainty measure, the proposed approach is also suitable for regression problems. For example, one may obtain the measure via predictive variance using kernel regression [41] or kriging [42] methods. Second, our method is motivated by active learning literature [29], where the next input is selected from a pool of candidates with maximal uncertainty. Different from those methods, our method (i) conducts sampling in a much lower-dimensional GIN feature space due to the intrinsic structure of image space and computational efficiency and (ii) sample a batch of images for labeling to both explore and exploit the design space. Moreover, it is worth pointing out that the proposed method is possible only when both the generating mapping $G(\cdot):\mathbb{F}\mapsto\mathbb{X}$ and the encoding mapping $E(\cdot):\mathbb{X}\mapsto\mathbb{F}$ are available via GIN. In particular, $G(\cdot)$ is used to generating images based on the selected features, while $E(\cdot)$ is used to embed the features of the actual images to guide the sampling. That is the key reason why we propose a bidirectional GIN in Section 3. Last but not least, the proposed method can also be used to balance the label distribution with a modification of our uncertainty measure (7). See Appendix F for more discussion.

A key component of the proposed AISEL framework, different from data augmentation, is the incorporation of physical knowledge while learning. This is due to the circumstances of real-world applications in manufacturing and healthcare: (i) the size of the historical records is small, leading to a poor learning model; and (ii) thanks to the advances in domain research, physical knowledge is oftentimes available yet expensive in implementation. Therefore, we want to build a bridge to efficiently combine both the historical data (via the learning model) and physical knowledge (via physical labeling). The resulting model can be viewed as one that has learned from data and been taught by physical knowledge, and therefore better performance can be achieved.

Here, we efficiently incorporate physical knowledge via a complementary AISEL dataset. Specifically, we separately acquire the input image and the output label. For virtual images, (10) is used to efficiently sample a set of features to minimize the predictive uncertainty, and GIN is then used to map those features to images. Meanwhile, for the labels, we use the physical labeling method at hand. We then combine the actual dataset and AISEL dataset to learn the downstream classification model. With the proposed uncertainty sampling method, our AISEL dataset (i) contains complementary information from physical knowledge, and (ii) efficiently exploit and explore the image space, and therefore improves the downstream learning performance.

Lots of different physics-based labeling approaches are available. For example, finite element analysis [43] and computational fluid dynamics [10] can solve partial differential equations (i.e., representation of physical knowledge) numerically. These methods can be used to label the input images in, e.g., manufacturing applications. Physical experiments can also be applied. With the advances in additive manufacturing, tissue-mimicking 3D printing [44] with an in-vitro study [45] can be used for medicine-related learning tasks. If none of the above exists, one can also consult the experts or use certain empirical physical relationships; these methods can be used in computer vision problems. The specific approach should be made on a case-by-case basis, with the available resources at hand.

It is important to note that labeling one input image via physical-based approaches is usually expensive. For example, in medicine-related applications, it may take several hours of computation for a CFD model with complex geometry [10], and it would be even longer if considering the interaction of blood flow and soft biological tissue [11]. This is one of the reasons for introducing an efficient and effective sampling method to design our AISEL dataset. Viewed this way, our approach can also be used to address the problem where an expensive simulator available, and we want to use that simulator actively for the classification tasks.

In summary, we propose an AISEL framework to efficiently incorporate physical knowledge at hand and improve the classification performance. The native model $C(\cdot)$ can be first obtained using the small training data. As illustrated in Fig.2, our AISEL framework contains three steps. First, the proposed GIN is trained using the actual images, providing a feature space $\mathbb{F}$, and bidirectional mappings between it and the image space (i.e., the generating mapping $G(\cdot)$ and the encoding mapping $E(\cdot)$). Second, the uncertainty of $C(\cdot)$ at different locations in $\mathbb{F}$ is quantified via entropy, and then the features for virtual images are sampled via (10). Third, virtual images are generated by $G(\cdot)$, and then labeled by the physics-based approach. Finally, the additional AISEL dataset is merged to the original training set, and an improved classifier $C^{*}(\cdot)$ can be trained. With the proposed AISEL framework to actively incorporate complementary knowledge via labeling, we will show later in the experiments, $C^{*}(\cdot)$ can achieve better classification accuracy.

We propose Algorithm 2 for our AISEL framework. In our implementation, the native model and improved model are parameterized by CNN, for its popularity in the image classification tasks. Other, perhaps more advanced architecture (e.g., ResNet [46]) can also be used. The optimization of (10) can efficiently implement by the convex-concave procedure (CCP, see [40]). Note that for all the NNs, especially the native model and the GIN, data augmentation methods (e.g., rotation and horizontal flip) are used. Note that our method can also be used for sequential implementation – run Algorithm 2 interactively to generate a series of AISEL datasets and therefore provide even better improvement, if budget allows.

We conduct experiments on small versions (400 in total for training) of two single-channel (i.e., grayscale) computer vision datasets – Fashion [47] and MNIST [48]. The two datasets are of particular interest due to their visual similarity to the images in the manufacturing process and modeling. For example, the images captured by a thermal camera (or simulated via finite element analysis), representing a gray-level temperature contour, can be used to predict the throughput in steel manufacture and conduct quality control in semiconductor manufacturing [49]. In the following subsections, we illustrate the visual performance of the proposed GIN, and the improvement in classification by our AISEL method, only on the Fashion dataset; similar observation also applies for MNIST (see Appendix E).

We now go back to the motivating application of aortic stenosis (AS). An anonymous image dataset containing 168 patients with aortic stenosis is collected (by Piedmont Healthcare, Atlanta). For each patient, pre-surgical CT scans and the corresponding calcification amount are acquired. The learning task is to classify the calcification level as high or low, which is an important yet challenging clinical problem. Four-fold cross validation strategy is used (see Appendix D), leading to only 126 data as the training set. We first provide more background information on the medical problem and our dataset. We then visualize the GIN performance, with a focus on the pathophysiological meaning. Finally, we discuss the classification accuracy, compared with baselines.