Proxyless Neural Architecture Adaptation for Supervised Learning and Self-Supervised Learning

To adapt the given neural architecture, we use differentiable architecture parameters and gradient-based learning. The architecture parameters $\theta$ is defined in the network architecture graph. Each edge in the network architecture graph contains original operation, identity operation, and none operation. Computation of each edge is carried out based on the architecture parameters:

where $x$ means input, ${o}_{e}(x)$ means output of the edge, $Z$ means zero tensor, $o(x)$ means original operation of the edge, ${\theta}_{e,none}$ means the weight of the none operation of the edge, ${\theta}_{e,id}$ means the weight of the identity operation of the edge, and ${\theta}_{e,same}$ means the weight of the original operation of the edge. We set initial ${\theta}_{none}$ and ${\theta}_{id}$ as zero, and ${\theta}_{same}$ as one. Therefore, the initialized network works the same as the original architecture.

After the network is initialized, the architecture train stage begins to adapt the neural architecture for the given dataset and task. In this stage, both the network weight parameters $\omega$ and the architecture parameters $\theta$ are trained alternately. For every input mini-batches, $\omega$ is trained first, and $\theta$ is trained after the update of $\omega$. Note that the proposed method doesn’t require any separated dataset for architecture optimization, and the network can utilize a full dataset to train its weights $\omega$. The architecture train stage is carried out for the pre-defined epochs.

When the architecture train stage is finished, architecture is transformed based on the trained $\theta$. For each edge, the operation that has the highest weight in $\theta$ is selected to construct the final architecture. In the example of Figure 1, two edges maintain the original operations, one edge changed its operation into identity, and one edge is removed because none operation is selected.

The architecture trained at the previous stage is fixed, and only $\omega$ of the network is trained in this stage. This stage is the same as the traditional neural network training process, and it is continuously carried out after the architecture train stage. At the end of this stage, we can get the trained network and use it to infer unseen input data or test the performance of the model.

The proposed method can be applied to both supervised learning and self-supervised learning. The objective of the proposed method can be formulated as:

where $\theta$ denotes the architecture parameter, $w$ is the weight of the network, $\mathcal{L}(\cdot)$ is the loss function.

In case of the supervised learning, we used the cross-entropy loss which uses data and labels to calculate the loss. In self-supervised learning experiments, we utilized SimCLR (Chen et al. 2020) objective which uses two different augmented views of images to calculate the loss.

Supervised Learning. CIFAR-10 dataset consists of 50,000 train images and 10,000 test images with ten classes. The size of images is $32\times 32$, and images have RGB color channels. Tiny Imagenet dataset has 100,000 train images and 10,000 test images with 200 classes. The input size of Tiny Imagenet dataset is $64\times 64$, and all images are RGB color images.

We experimented with various models on CIFAR-10 and Tiny Imagenet dataset. These models include ResNet20, MobileNet V2, DARTS, and ProxylessNAS. The former two models are manually designed, and the latter two models are NAS models. To compare the performance of NAT and our algorithm, we trained the NAT controller on each dataset and then trained the transformed architecture inferred from the controller. In the case of our algorithm, we test both cell-based transformation and full network transformation. We used 0.025 learning rate, 600 epoch, and Stochastic Gradient Descent(SGD) optimizer as the same hyper-parameters to all models and methods. Exceptionally, we applied 300 epochs to Mobilenet V2 and DARTS on Tiny Imagenet dataset, and utilized cut-out for NAS models such as DARTS and ProxylessNAS.

We tested five times with different random seeds for every experiment to get the right performance and verify the reproducibility of comparison algorithms. Therefore, we report the average accuracy and standard deviation of each method and each model. The total cost of NAT was calculated by adding GPU hours of the architecture transformation stage and network train stage, and the cost of our algorithms was computed by just measuring the cost of the whole training process.

The results of Table 1 have average accuracy, standard deviation, and total cost by various methods with different models on CIFAR-10 dataset. We trained and inferred five times to get average accuracy and standard deviation. As shown in Table 1, the results of NAT is unstable in the case of manually designed models. The results of our algorithms have better average accuracy and standard deviation than original and NAT in all cases. Moreover, the total computational cost is lower than NAT. Note that NAT cannot transform the architecture of ProxylessNAS, since it has various cell architectures in the network. However, the proposed method successfully improves the performance of ProxylessNAS architecture.

Table 2 shows the reproducibility of various methods with different random seeds on CIFAR-10 dataset. Only one result is presented for each model in NAT paper. Therefore we experimented five times to get the right performance. In the case of Resnet20 experiments, the results of seed 1 and 4 of NAT are caused by transforming identity edges into none operation. Transformed Resnet20 architectures of seed 1 are presented in Figure 2. Changed edges are notated as red colors. As shown in Figure 2(b), NAT transformed all edges to node 5 into none operation. Therefore, zero tensors are passed to the next layer.

Regarding the result of Mobilenet V2 experiments, the performance of NAT is degraded when it transforms convolution operation into identity operation. Figure 3 shows the transformed Mobilenet V2 architectures of seed 1. Additionally, we represent the transformed DARTS normal cell architecture of our algorithm in Figure 4. There is no change in the edges of the reduction cell.

Table 3 shows the results of various methods with different models on Tiny Imagenet dataset. The results of this table describe that both our algorithms have better average accuracy and standard deviation than the original method upon all models.

The results reported in Table 4 is evaluated by k-nearest neighbor (KNN) during the representation learning. Therefore, it may not be better than the performance of linear evaluation reported in (Chen et al. 2020) but we evaluated all architectures fair conditions. According to the results of Table 4, the proposed method outperforms the basic architecture of ResNet50 and DARTS even with similar parameter sizes.

In ResNet50, three operations were changed from none to skip-connection compared to the original architecture. And in DARTS, skip-connection was changed to none in only two places in the reduce cell, resulting in little difference in the size of parameters compared to the cases of supervised learning. This is due to the fact that self-supervised learning starves a lot of parameters than supervised learning.