Robust Neural Architecture Search

In the past few years, NAS has become more and more popular because it can take a little consumption cost to design neural network architectures with good performance[10]. However, NAS-generated models tend to suffer greater vulnerability to various malicious attacks (e.g., adversarial evasion, model poisoning, and functionality stealing) [5]. Lots of methods have been proposed to improve the robustness of NAS. RobNet [8] proposed to leverage adversarial training to improve the robustness. RACL [11] proposed to reduce the Lipschitz constant to improve the robustness of NAS. These methods are either too complex or only concerned on robustness. In our paper, we introduce a simple regularizer to consider the correlation between the output of the supernet on natural data and adversarial data and use the regularizer to balance accuracy and robustness.

Adversarial attacks are to generate adversarial data to attack models. There are some kinds of adversarial attacks, including evasion attacks [12], poisoning attacks [13], and so on. In the paper, we mainly discuss evasion attacks. Evasion attacks attach imperceptible perturbation $\delta$ on input $x$ to generate an adversarial input $x+\delta$. Then, the adversarial input $x+\delta$ is used to deceive the model to miscategorize input $x$ into other classes. Thus, it is important to generate perturbation $\delta$, and the problem can be formulated as follows:

where $\mathcal{B}$ is represented as the perturbation set.

The aim of adversarial defense methods is to enhance the robustness of the models so that defense adversarial attacks, and there are lots of adversarial defense methods. Adversarial training is a kind of adversarial defenses methods. Adversarial training is to generate adversarial examples and use these adversarial examples to train the model to enhance its robustness of the model. PGD [6] and its variant [14, 15] are used to solve the min-max problem and get good results. However, Tsipras et al. [7] proposes that there is a trade-off between accuracy and robustness, thus, high accuracy and good robustness cannot be both achieved. To solve the problem, it is a good idea to use NAS to design the networks with high accuracy and robustness. Lots of works [8, 9] have been proposed to prove the feasibility of this idea. In our paper, we also try to combine NAS and adversarial training to search architectures with good performance.

At first, we make a brief introduction about adversarial training and neural architecture search(NAS).

Adversarial Training. Deep neural networks (DNNs) are vulnerable to perturbs on examples, and the performance of these models becomes bad under attack. Thus, adversarial training [16] is proposed to enhance the robustness of DNNs, it uses adversarial examples instead of natural examples as input to train DNNs. The problem of adversarial training can be formulated as a minimax optimization problem:

where $f_{\theta}(\cdot)$ denotes the model, $D$ denotes the data distribution and $\left\|x^{\prime}-x\right\|\leq\epsilon$ defines the set of allowed perturbation inputs. Then, Madry et al. [6] leverages Projected Gradient Descent (PGD) to solve the problem to generate adversarial examples and use these examples as input to train the model to enhance the robustness. In our work, we also leverage adversarial examples to search those robust architectures.

Neural Architecture Search. Neural Structure Search (NAS) is a technology to design neural network architecture automatically. NAS consists of three parts: search space, search strategy, and performance estimation. The search space includes a set of candidate neural networks that can be searched. The search strategy defines the method to find the optimal candidate network. The performance estimation strategy aims to evaluate the performance of the sampled network. The main purpose of NAS is to find the optimal network from the search space, and it can be represented as a bi-level optimization problem:

where $\alpha$ denotes the candidate network and $w$ denotes the parameters of the networks. Recently, Differentiable Architecture Search (DARTS) [1] as a very popular NAS method due to low computing consumption and extremely fast search speed. In our work, we will leverage DARTS as a representative NAS method to search robust architectures.

The architectures searched by NAS under nature examples have the better accuracy, however, they are not robust and vulnerable to adversarial examples. On the other hand, the architectures searched by NAS under adversarial examples have high robustness, but there was a decrease in their accuracy. Our aim is to search architectures with high accuracy and good robustness.

In our paper, we use model prediction performance under natural examples as the metric to search the architectures with good accuracy. On the other hand, we propose a regularization term to search the robust architectures, i.e., maximize adversarial output similarity under nature examples and adversarial examples. Thus, the NAS optimization problem can be formulated as follows:

where $m$ denotes the supernet and $\lambda$ as a regularization coefficient controls the tradeoff between robustness and accuracy. When $\lambda$ becomes larger, more attention will be paid to robustness than accuracy. We call our method RNAS-max simply.

Fig. 1 introduces the main framework of RNAS-max. PGD [6] is a mainstream optimization method to generate adversarial examples. Thus, we also leverage PGD to optimize RNAS-max, and the algorithm of RNAS-max is represented as Alg. 1. At first, we initialize architecture parameters $\alpha$ and network parameters $w$. Then, we use valid data as input and leverage PGD to optimize $\mathop{argmax}\limits_{x^{\prime}\in S}f(m(x^{\prime}),m(x))$ to generate adversarial valid examples. After adversarial valid examples are generated, we leverage adversarial valid examples and natural valid examples as input to put in the supernet to compute their outputs. Then, we compute the correlation of two outputs and nature valid loss and take the sum of them as a robust valid loss. We leverage Adam to optimize robust valid loss to get the optimal architecture parameters $\alpha$. Similar to the DARTS, we alternate optimizing the architecture parameters $\alpha$ and the network parameters $w$. Thus, we use training data as input to generate adversarial training examples by optimizing $\mathop{argmax}\limits_{x^{\prime}\in S}f(m(x^{\prime}),m(x))$. Then, we use adversarial training examples and natural training examples as the input of the inner optimization problem (Eq. 4) and use SGD to optimize the inner optimization problem to get the optimal network parameters $w$. We use this way to optimize the problem (Eq. 4) several steps until get the optimal architecture parameters $\alpha$ and the network parameters $w$. Finally, we derive the optimal subnet with good robustness and high accuracy.

Section 3.2 proposes to add a regularization term to the bi-level optimization problem of NAS to generate neural network architectures with higher quality. Although the method is effective, it is not efficient enough. Eq. 4 becomes a three-level optimization problem, and the computational complexity of the three-level optimization problem (Eq. 4) is far higher than the bi-level optimization problem (Eq. 3). It will take several times longer than standard DARTS to generate the optimal subnet. The problem is a serious obstacle to the practical application of our method. In our paper, we try to generate some noise examples instead of adversarial examples to train RNAS for reducing consumption cost.

We think that robust models are also insensitive to random noise in samples, and the supernet is the set of lots of subnets. To make the supernet insensitive to random noise in samples, these robust subnets will have bigger architecture weights than others. We sample some random examples from the perturbation set as noise examples instead of adversarial examples. We call the method RNAS-uniform simply. Then, the RNAS-uniform can be formulated as follows:

where $x^{\prime}$ is represented as noise examples, and $U$ is represented to sample a perturbation example uniformly from the perturbation set. Eq. 5 becomes an bi-level optimization problem, and we only need to spend a little extra consumption cost to generate noise examples. Although we leverage noise examples to train NAS to generate architectures with worse robustness and lower accuracy than adversarial examples, it can reduce search time a lot.

Fig. 2 introduces the main framework of RNAS-uniform, and the algorithm of RNAS-max is represented as Alg. 2. At first, we generate perturbation valid examples by sampling uniformly from the perturbation valid set. Then, we use the perturbation valid examples and the natural valid examples as input, and update architecture parameters $\alpha$ by optimizing $\mathop{min}\limits_{\alpha}\mathcal{L}_{val}(w^{*}(\alpha),\alpha)+\lambda\mathop{U}\limits_{x^{\prime}\in S}f(m(x^{\prime}),m(x))$. We also alternate optimizing architecture parameters $\alpha$ and network parameters $w$ as DARTS. Thus, we generate perturbation training examples by sampling uniformly from the perturbation training set. Afterwards, we use the perturbation training examples and the natural valid examples as input, and update network parameters $w$ by optimizing $\mathop{argmin}\limits_{w}[\mathcal{L}_{\text{train }}(w,\alpha)+\lambda\mathop{U}\limits_{x^{\prime}\in S}f(m(x^{\prime}),m(x))]$. We use the way to optimize the problem (Eq. 5) several steps. Finally, we derive the optimal subnet based on architecture parameters.

CIFAR-10 includes 50K training images and 10K testing images, and each image’s spatial resolution is $32\times 32$. These images belong to 10 classes. When searching the architecture, we divide these training images into two subnets, i.e., the training subnet and the valid subnet. The training subnet is used to optimize network parameters, and the valid subnet is leveraged to optimize architecture parameters.

DARTS is one of the most popular NAS methods, and we use DARTS as a representative method to combine our method to search the architecture with good performance. Similar to standard DARTS, we aim to search for a normal cell and a reduction cell to build a network with good performance on the image classification task. Our method has two submethods, i.e., RNAS-max and RNAS-uniform, and we introduce the experiment settings of the two submethods as follows.

We introduce the experiment setting of RNAS-max at first. Most of our experiment settings are consistent with DARTS, and we make a few modifications to others. We run rNAS for a total of 50 epochs to search the architecture, namely total epochs $T$ for search is 50. The batch size $m$ of our experiment is 64. The regularization coefficient $\lambda$ is set as 1.0. The neural network at initialization does not contain valuable information, so we first warm up the neural network for 15 epochs, i.e., epochs $W$ for warm-up is 15. We use PGD to generate adversarial examples. In our experiment settings, we use 10-PGD to generate adversarial examples, i.e., num steps $K$ is 10. Further, the step size $\eta$ of 10-PGD is 0.003, and the total perturbation scale $\epsilon$ is 0.031. We use momentum SGD to update network parameters $w$, and its initial learning rate is 0.025, momentum is 0.9, and weight decay factor 3e-4. Then, we use Adam to update architecture parameters $\alpha$, and its initial learning rate is 3e-4, momentum is $(0.5,0.999)$, and weight decay factor is 1e-3. We spend 4.3 GPU days generating the architecture.

The experiment settings of RNAS-uniform are also consistent with DARTS and RNAS-max mostly. The total epochs $T$ of RNAS-uniform are also 50, including 15 epochs $W$ for warm_up. The batch size $m$ and regularization coefficient $\lambda$ of RNAS-uniform are also consistent with RNAS-max, i.e., 64 and 1.0. We sample uniformly noise examples from the perturb set, and the perturbation bound $\epsilon$ is 0.031. The optimizers to optimize architecture parameters $\alpha$ and network parameters $w$ are consistent with RNAS-max, and the hyperparameters of the two optimizers are also the same as RNAS-max. We spend 0.5 GPU days to generate our architecture, and the search time is only slightly slower than DARTS.

We use these cells searched by RNAS to build networks and evaluate the performance of these networks on CIFAR-10. We use 20 cells to build a network, including two reduction cells located at 1/3 and 2/3 of the total depth of the network. We run the training process 600 epochs, and the batch size of our training set is 96. We also use SGD to optimize the network, and the initial learning rate is 0.025. We also cutout and auxiliary towers to help us train the networks; the length of the cutout is 16, the weight of auxiliary towers is 0.4, and the probability of path dropout is 0.3.

Table 1 summarizes evaluation results and comparison with state-of-the-art approaches. Table 1 shows that RNAS-max achieves a 2.65% test error on CIFAR10 with a search cost of 4.3 GPU-days. Although the search time of RNAS-max is beyond DARTS a lot, the performance is far better than standard DARTS; the params of the architecture generated by RNAS-max is 3.4M, and it is a little larger than the architecture searched by DARTS and smaller than the architecture searched by some variants of DARTS, e.g., SGAS [28], PC-DARTS [24]. RNAS-uniform achieves a 2.60% test error on CIFAR-10 with a search cost of 0.5 GPU-days. The performance of RNAS-uniform is far better than standard DARTS, while the search time of RNAS-uniform is 0.5 GPU days, less than DARTS ($2^{\text{nd}}$ order), and a little more than DARTS ($1^{\text{st}}$ order). The parameters of the architecture searched by RNAS-uniform are also 3.4M, and the architecture is lightweight enough to be run on mobile devices. In general, RNAS makes a big improvement on DARTS, and can search good architectures with high accuracy and good robustness.

We evaluate the robustness of architectures that are standard and adversarially trained on CIFAR-10 by using lots of adversarial attacks. Standard training is to leverage natural examples to train the networks. The experiment settings of standard training are the same as DARTS. Adversarial training is to use adversarial examples to train the networks. In our paper, we use 7-step PGD to train the networks; the step size is 0.01 and the perturbation scale is 0.031. We use FGSM to attack standard-trained architectures; FGSM and PGD²⁰ are used to attack adversarially-trained architectures. Table 2 shows the evaluation results on CIFAR-10 under adversarial attacks. RNAS-max achieves good robust accuracy by standard training or adversarially training, it gets the highest robust accuracy than other methods. However, RNAS-uniform has bad performance on robustness, and it means that RNAS-uniform sacrifices the robustness of the architecture and has a little effect on accuracy by using noise examples as input. In the further, we seek a better method to improve the robustness and accuracy of the architectures searched by NAS with low consumption cost.