Exploring Complicated Search Spaces with Interleaving-Free Sampling

In neural architecture search (NAS), a deep neural network can be formulated into a mathematical function that receives an image $\mathbf{x}$ as the input and produces the desired information (e.g., a class label $y$) as the output. We denote the function to be $y=f(\mathbf{x};\boldsymbol{\alpha},\boldsymbol{\omega})$ where the form of $f(\cdot)$ is determined by a set of architectural parameters, $\boldsymbol{\alpha}$, and the learnable weights (e.g., the convolutional weights) are denoted by $\boldsymbol{\omega}$. The goal of NAS is to find the optimal architecture, $\boldsymbol{\alpha}^{\star}$, that leads to the best performance, i.e.,

		$\displaystyle{\boldsymbol{\alpha}^{\star}}={\arg\min_{\boldsymbol{\alpha}}\mathbb{E}_{\left(\mathbf{x},y^{\star}\right)\in\mathcal{D}_{\mathrm{val}}}\!\left|y^{\star}-f(\mathbf{x})\right|}$			(1)
	$\displaystyle\mathrm{s.t.}$	$\displaystyle{\boldsymbol{\omega}^{\star}(\boldsymbol{\alpha})}={\arg\min_{\boldsymbol{\omega}}\mathbb{E}_{\left(\mathbf{x},y^{\star}\right)\in\mathcal{D}_{\mathrm{train}}}\!\left|y^{\star}-f(\mathbf{x})\right|},$

where $y^{\star}$ denotes the ground truth label. Most often, $\boldsymbol{\alpha}$ takes discrete values, implying that solving Eqn (1) requires enumerating a large number of sampled architectures and performing individual evaluation. To accelerate, researchers propose to slack $\boldsymbol{\alpha}$ into a continuous form so that solving Eqn (1) involves optimizing a super-network, after which $\boldsymbol{\alpha}^{\star}$ is discretized into the optimal architecture for other applications.

In particular, this paper is built upon the differentiable search algorithms, in which the super-network is solved by computing the gradient with respect to $\boldsymbol{\alpha}$. We will introduce the details of optimization in the experimental part.

We design a search space shown in Figure 2. We define a fixed integer, $L$, indicating that each layer can be connected to $L$ precursors. For convenience, we name this space $L$-chain space. When $L=1$, it degrades to the chain-styled network (the backbone of MobileNet-v3). On the contrary, we study the cases of $L=4$, $L=6$, and even $L=8$, making the topology of the space much more complicated. We follow R-DARTS [24] to allow two candidate operations, i.e. the $3\times 3$ separable convolution and skip-connection. Throughout the remaining part of this paper, we use $\mathbb{S}^{(L)}$ to indicate the proposed space with $L$ precursor connections. For convenience, the connection between the $n$-th and $(n-l)$-th layers (nodes) is named the pre-$l$ connection of the $n$-th layer.

Before entering the algorithm part, we emphasize that we do not use additional rules to assist the architecture search, e.g., forcing all nodes to survive by preserving at least one connection. This raises new challenges to the search algorithm, because the depth of the searched architecture becomes quite indeterminate. Although some prior work has explored the option of optimizing the macro and micro architectures jointly [15] or adding mid-range connections to the backbone [8], they still rely on the cell/block unit to perform partial search. Instead, we break the limitations of the unit, and allow search in a wider and macro space.

In each edge, there are two candidate operators in our $L$-chain space. Each node receives at most $2L$ input feature maps. For the purpose of validity, each node has at least one input to be preserved. However, not every node may be used as an input node, and these nodes will be deleted, which means that the architectures are allowed to be very shallow. For a node $x_{k}(k>L)$, there are $2^{2L}-1$ input possibilities because each of the $2L$ operators can be on or off. As a result, there are $(2^{2}-1)\times(2^{4}-1)\times\cdot\cdot\cdot\times(2^{2(L-1)}-1)\times(2^{2L}-1)^{N-L}$ combinations if there are $N$ nodes in one stage. There are 3 stages to be searched in our space, Fig. 2, with the node number of 18, 20 and 18 respectively. Therefore, if $L$ is 4, 6 and 8, there are about $1.8\times 10^{116}$, $7.5\times 10^{163}$ and $1.6\times 10^{204}$ possible architectures respectively. The complexity comparison of popular spaces are shown in Tab. 1.

We first evaluate DARTS [17] and GOLD-NAS [3] in the search space of $\mathbb{S}^{(4)}$. On the ImageNet-1k dataset, trained for 250 epochs for each network, DARTS and GOLD-NAS report top-1 accuracy of $74.7\%$ and $75.3\%$, respectively, and the FLOPs of both networks are close to $600\mathrm{M}$ (i.e., the mobile setting), Tab. 2. In comparison, the simple chain-styled architecture (each node is only connected to its direct precursor) achieves $74.9\%$ with merely $520\mathrm{M}$ FLOPs. More interestingly, if we only preserve one input for each node, DARTS and GOLD-NAS report completely failed results of $70.2\%$ and $71.2\%$ (trained for 100 epochs), which are even much worse than preserve one input randomly, Tab. 3.

We investigate the searched architectures, and find that both DARTS and GOLD-NAS tend to find long-distance connections. This decreases the depth of the searched architectures, however, empirically, deeper networks often lead to better performance. This drives us to rethink the reason that the algorithm gets confused. For this purpose, we randomly choose an intermediate layer from the super-network and the algorithm needs to determine the preference among its precursors.

We start from the simplest situation that all network connections are frozen (the architectural weights are fixed) besides the candidate connections. We show the trend of three (pre-$1$, pre-$3$, pre-$6$) connections in Figure 1. One can observe that the pre-$1$ connection overwhelms other two connections, aligning with our expectation that a deeper network performs better. However, when we insert one connection that lies between these candidates, we observe that the advantage of the pre-$1$ connection largely shrinks, and a long training procedure is required for it to take the lead. The situation continues deteriorating if we insert more connections into this region. When two or more connections are added, the algorithm cannot guarantee to promote the pre-$1$ connection and, sometimes, the ranking of the three connections is totally reversed.

The above results inspire us that two connections are easily interfered by each other if the covering regions (i.e., the interval between both ends) overlap. Hereafter, we name such pair of connections interleaved connections. Mathematically, denote a connection that links the a-th and b-th nodes as $(a,b)$, where $b-a\geq 2$. Overlook the candidate connections of the activated nodes, two connections, $(a,b)$ and $(a^{\prime},b^{\prime})$, interleave if and only if there exists at least one integer $d$ that satisfies $a<d+1/2<b$ and $a^{\prime}<d+1/2<b^{\prime}$. Intuitively, for a real number $x$, if $\{x|a<x<b\}\cap\{x|a^{\prime}<x<b^{\prime}\}\neq\emptyset$ is satisfied, the interleaved connection occurs.

As a side comment, the DARTS space is also impacted by the interleaved connections, which partly cause the degradation problem observed in prior work [14, 6, 20]. However, since the search space is cell-based, the degradation does not cause dramatic accuracy drop. In the experiments, we show that our solution, elaborated in the next part, generalizes well to the DARTS space.

Following the above analysis, the key to design the search algorithm is to avoid interleaved connections, meanwhile ensuring that all connections can be considered. For this purpose, we propose IF-NAS, a sampling-based approach that each time optimizes a interleaving-free sub-super-network from the super-network.

This is implemented by partitioning the layers into $L$ groups, $\mathcal{G}_{1},\mathcal{G}_{2},\ldots,\mathcal{G}_{L}$, according to their indices modulo $L$. In other words, the distance between any consecutive layers that belong to the same group is exactly $L$. When any group $\mathcal{G}_{\lambda}$ is chosen, we obtain an interleaving-free sub-super-network by preserving (i) the main backbone (all pre-$1$ connections) and (ii) all connections that end at any layer in $\mathcal{G}_{\lambda}$. We denote the sub-super-network by $\mathbb{S}_{\lambda}^{(L)}$ for $\lambda=1,2,\ldots,L$. Correspondingly, the optimization goal can be written as:

$$\min_{\boldsymbol{\alpha},\boldsymbol{\omega}}\mathbb{E}_{\left(\mathbf{x},y^{\star}\right)\in\mathcal{D}}\left|y^{\star}-f\!\left(\mathbf{x}\mid\mathbb{S}_{\lambda}^{(L)}\right)\right|.$$

(2)

A straightforward search procedure repeats the loop of $L$ sub-super-networks, as shown in Figure 3. Training each sub-super-network is an approximation of optimizing the entire super-network, $\mathbb{S}^{(L)}$, but, since the sub-super-network are fixed, the co-occurring connections may gain unexpected advantages over other ones. To avoid it, we add a warm-up step to the beginning of each loop, in which the entire super-network is trained (all connections are activated, but the architectural parameters are not updated). Throughout the remainder of the loop, we train the $L$ sub-super-networks orderly. We set the length of each step to be $100$ iterations (i.e., mini-batches). Note that this number shall not be too large, otherwise the gap between the sampled and non-sampled connections will increase and harm the search performance.

The last step is to determine the final architecture. In a complicated search space, this is not simple as it seems, because many connections may have moderate weights (i.e., not close to $0$ or $1$). For such a connection, either pruning it or promoting it can bring significant perturbation on the super-network. Therefore, we follow the idea of prior works [7, 3, 21] to first perform a discretization procedure to push the weights towards either $0$ or $1$, after which pruning is much safer. We introduce two kinds of architectural parameters, the connectivity of edges are determined by $\boldsymbol{\beta}$ and operators are determined by $\boldsymbol{\alpha}$.

The key of discretization is to add a regularization term to Eqn. (2). The term penalizes the moderate weights of edges and operators, represented by $\boldsymbol{\beta}$ and $\boldsymbol{\alpha}$ respectively. Once these weights are less than $0.01$, the corresponding edges or operators will be pruned off. The regularization term is computed by:

$$\begin{split}\mathcal{R}(\alpha,\beta)&=\mu_{1}\cdot\sum_{j\leq N}\sum_{0\leq j-L\leq i<j}\ln(1+g(\beta_{i,j})/\overline{g(\beta_{i,j})})\\ &+\mu_{2}\cdot\sum_{j\leq N}\sum_{0\leq j-L\leq i<j}\sum_{o\in\mathcal{O}}\ln(1+g(\alpha^{o}_{i,j}|\beta_{i,j})/\overline{g(\alpha^{o}_{i,j}|\beta_{i,j})}),\end{split}$$

(3)

where $N$ is the number of layers, and $g(\cdot)$ is the activate function $sigmoid$. $g(\alpha|\beta)$ means to only include operators on $\boldsymbol{\beta}$ that have not been pruned off. $\overline{g(\cdot)}$ is the average of $g(\cdot)$. $\mathcal{R}(\alpha,\beta)$ is multiplied by a factor of $\mu$ and added to the cross-entropy loss, so that optimizing the overall loss will not only improve the recognition accuracy of the super-network, but also push the weights of all connections towards $0$ or $1$.

When the weight of a connection is sufficiently small, we prune it permanently from the super-network. This merely impacts the super-network itself, but the computation of $\overline{g(\cdot)}$ changes and will push the next weak operator to $0$. This process iterates until the complexity (e.g., FLOPs) of the super-network achieves the lower-bound.

The Dataset. We use the large-scale ImageNet dataset (ILSVRC2012) to evaluate the models. ImageNet contains 1,000 object categories, which consists of 1.3M training images and 50K validation images. The images are almost equally distributed over all classes. Unless specified, we apply the mobile setting, in which the input image is set to $224\times 224$ and the number of multiply-add operations (MAdds) does not exceed 600M. We randomly sample 100 classes from the original ImageNet dataset to perform studying and analysis experiments, ImageNet-100 for short.

The Search Settings. Before searching, IF-NAS warm-ups the super-network for 20 epochs, with only the super-network weights updated and the architectural parameters frozen. Then we start to update $\beta$ firstly, and after 10 epochs, start to update $\alpha$. Super-network weights and architectural parameters are both optimized by the SGD optimizer. We gradually prune off weak edges and operators with threshold of $0.01$ until the MAdds of the retained architecture meets the mobile setting, i.e., 600M.

The Training Settings. We evaluate the searched architectures following the setting of PC-DARTS [23]. Each searched architecture is trained from scratch with a batch size of 1024 on 8 Tesla V100 GPUs. A SGD optimizer is used with an initial learning rate of 0.5, weight decay ratio of $3\times 10^{-5}$, and a momentum of 0.9. Other common techniques including label smoothing, auxiliary loss, and learning rate warm-up for the first 5 epochs are also applied.

We study our method IF-NAS on the proposed macro $L$-chain search space, and compare it with three other popular DNAS frameworks, including DARTS [17], PC-DARTS [23] and GOLD-NAS [3]. For a fair comparison, the hyper-parameters are kept the same for all the studied methods. Without pruning gradually, DARTS and PC-DARTS derive the final architectures by removing weak edges and operators after searching until they meet the requirements of mobile setting. All the searched architectures are re-trained from scratch on ImageNet-1k for 250 epochs.

We perform three sets of experiments by setting $L$, which indicates the precursors number of each layer can be connected, to 4, 6 and 8 respectively, and show the results in Tab. 2. When $L$ is 4, DARTS gets the worst performance as expected. Compared with DARTS, PC-DARTS and GOLD-NAS get better results of $24.7\%$ and $24.7\%$ with similar MAdds. Our IF-NAS gets the best result of $24.3\%$ with equivalent Params and MAdds. This proves that IF-NAS can work better in such a enlarged complicated space compared to the methods.

When $L$ is 6, DARTS achieves the result of $25.7\%$. PC-DARTS and GOLD-NAS get results of $24.9\%$ and $25.0\%$ respectively. And our IF-NAS gets the best result of $24.4\%$ with similar Params and MAdds. Note that the results of the compared 3 frameworks get worse when $L$ increases to 6 compared to $L$ as 4. However, IF-NAS nearly maintains the performance. When $L$ increases, more interleaved connections occur, and bring more interference to the search process. The compared 3 methods, which do not suppress interleaved connections during search, are more severely effected and achieve worse results. However, IF-NAS samples interleaving-free sub-networks and updates the architectural parameters more accurately. This further confirms that our IF-NAS has great potential to handle complicated spaces.

When set $L$ to 8, IF-NAS still achieves a promising result of $24.3\%$, which is comparable to the former two settings. The compared ones get similar or weaker results. Due to the increase in complexity, the performances of PC-DARTS and GOLD-NAS continue to decrease. But the MAdds of these architectures are close to 600M, which prevents performance from declining.

If we do not derive architectures with MAdds and preserve one input for each node, the results are shown in Tab. 3. Interestingly, without the protection of keeping so many MAdds, DARTS and GOLD-NAS show completely failed results. When $L$=4, the results of DARTS and GOLD-NAS are $29.8\%$ and $28.8\%$, which are even much worse than the randomly generated architectures. When $L$ increases to 8, the performances of DARTS and GOLD-NAS further degrade. The architectures searched without suppressing interleaved connections only preserve very few nodes, which damages the final performance dramatically. On the contrary, IF-NAS still achieves comparable results even without any MAdss restriction.

The above experiments imply interleaving connections cause non-negligible impacts to DNAS methods when the search space grows larger and more complicated. IF-NAS is able to handle this challenge by the proposed interleaving-free sampling.

In this part, we design a comparative experiment to verify the effectiveness of IF sampling. We search for 4 times independently for each setting, and the results are shown in Tab. 4. The Depth in Tab. 4 refers to the depth summation of 3 stages, which represents the longest path length of the network. And all architectures are train on ImageNet-1k for 100 epochs.

We find the accuracy of the architectures searched without IF (“w/o IF”) sampling is much worse than that with IF (“w/ IF”) sampling by a margin of $0.7\%$ with similar Params. It is worth noting that the depth of w/o IF is much smaller than w/ IF, because the interleaved connections produce serious interference to the search process. This verifies that our IF-NAS can effectively handle interleaved connections and guarantee the search performance.

The visualizations of searched architectures are shown in Fig. 4 and Fig. 5. We can find that w/ IF preserves many serial pattern connections and w/o IF mainly preserves parallel connections, as a result, the architectures of w/ IF is much deeper than w/o IF. Fig. 6 shows some weight change curves of w/ IF and w/o IF. One can see that w/ IF can select more closer inputs, which derives deeper architectures. w/o IF prefers farther nodes, which usually leads to parallel connections and discards more nodes.

We allow various numbers (1/2/3) of interleaved connections to be sampled during search to understand the effects they may bring. The searched architectures are trained on ImageNet-100 for 100 epochs and the results are shown in Tab. 5. We find with more interleaved connections added, the performance degrades more. This shows more interleaved connections cause greater impact on the search. It implies that Interleaving-Free is necessary in such a flexible space. Interestingly, the performance drops greatly even through one interleaved connection is sampled during searching, which demonstrates that as long as the interleaved connection is involved, interference is generated.

In this section, we test IF-NAS in two micro search spaces, DARTS space referred to as $space_{D}$ and GOLD-NAS space referred to as $space_{G}$. The two spaces adopt the same $3\times 3$ separable convolution and skip-connection operators as ours, thus that the comparison is fair. The results are shown in Fig. 7. The horizontal axis denotes the space complexity (the exponential base of 10) and the vertical axis denotes the accuracy trained on ImageNet-1k for 100 epochs. The bubble sizes imply the complexity of spaces. The results on $space_{D}$ and $space_{G}$ are close among 3 strategies (even with random search), which implies the 2 spaces are inflexible and have a protection mechanism (the cell design) to help the search methods. The $L$-chain spaces are more flexible and difficult so that the gaps between IF-NAS and others in $L$-chain spaces are much larger. Moreover, the larger the spaces, the worse the performances of DNAS and random search. However, our IF-NAS provides all best results in the 5 spaces, which shows that IF-NAS are robust and generalize well to micro spaces.

We also test our IFNAS by training 250 epochs under the mobile setting and the results are shown in Table 6. IFNAS achieves $24.2\%$ test error in $space_{D}$, which is same as PCDARTS (a specially designed method in this space). In $space_{G}$, our IFNAS-G1 beats GOLD-I by $0.2\%$ with $0.2$M fewer parameters and our IFNAS-G2 surpasses GOLD-X with $0.4$M fewer parameters.

In Section 3.3, we show failure cases from the simplest situation. Here we perform more experiments to show the influence of interleaved connections and the weight change curves are shown in Figure 8. In Fig. 8 (a), we run 4 times independently under interleaving-free setting and the nearest candidate inputs have a dominant weight at this time. In Fig. 8 (b)-(d), We run 4 times independently and each time we add 1-3 different interference connections. One can see that the more interleaved connections are added, the harder it it the nearest candidate input is picked.