Competition-based Adaptive ReLU
for Deep Neural Networks

Our proposed method stems from the idea that both positive values and negative values are equally important and they must compete for activation. We choose energy as an indicator of the competition. The rule is that the competitor with higher energy wins the qualification for activation. Let $\bm{z}=(z_{1},z_{2},...,z_{d})\in\mathbb{R}^{d}$ be the output of the previous affine transformation. The percentage of positive values’ energy $p_{E}$ is defined as follows:

where a small positive constant value $\epsilon$ is added in the denominator to prevent the division by zero. We define $s$ as follows:

where ${\rm sgn}(\cdot)$ is the sign function. If $p_{E}<0.5$ is true, then we have $s=-1$, which means the negative values win the competition; If $p_{E}>0.5$ is true, then we have $s=1$, which means the positive values win the competition. By multiplying $\bm{z}$ with $s$ when passing it to the ReLU, we can flip the input if negative values have higher energy:

However, this vanilla version of our idea does not perform well. First, there is no degeneracy into regular ReLU when prioritizing higher energy impairs performance. Second, the sign function creates discontinuities in the model’s parameter space, which harms the gradient-based optimization. To address the first issue, we replace the fixed scaling factor of $2$ and the bias of $-1$ in Eq. (4) with trainable parameters $\alpha$ and $\beta$ per layer:

If $\alpha$ is close or equal to $0$ after training, this mapping degenerates into ReLU with a factor of $1$ or $-1$. Only 2 extra parameters are introduced per layer, which is negligible when considering the total number of weights.

To address the second issue, we replace the sign function with the tanh function as follows:

The tanh function has a codomain ranging from $-1$ to $1$ and it is a continuous function with a non-zero gradient, which can be viewed as a smooth version of the sign function.

There are other kinds of indicators for competition between positive values and negative values other than energy. The other two indicators we experiment with are:

where $H(\cdot)$ is the Heaviside step function, Eq. (8) is the percentage of positive values’ L1-norm, and Eq. (9) is the percentage of the number of positive values.

Competition-based Adaptive Scaling (CAS) can be defined as follows to adaptively scale the input $\bm{z}$ based on competition results:

where $K$ is a constant and $p\in\{p_{E},p_{L1},p_{c}\}$. Since $|\tanh(\alpha p+\beta)|$ is less than 1, the magnitude of feature vectors will shrink layer by layer in a sequence model. To combat this phenomenon, the constant $K$ is placed to scale up the results of the tanh function. By concatenating CAS and ReLU, we construct Competition-based Adaptive ReLU as follows:

Batch Normalization (BN) [18] is widely used in convolutional neural networks. BN normalizes and rescales the input with 2 parameters on a mini-batch before activation. Since CAReLU requires competition between positive values and negative values, the BN’s normalization might impair this process. To improve our method’s compatibility with BN, we define $\rm BN\text{-}CAReLU$ as follows by placing Eq. (10) before BN so that the neural network can obtain competition results before the normalization:

Let $\alpha_{0}$ be the initial value of $\alpha$ and $\beta_{0}$ be the initial value of $\beta$. Instead of running a grid search for initial values, we simply set these values as follows:

where $\alpha_{0}=0$ means that deep neural networks don’t utilize competition results at the beginning. $\beta_{0}=1$ makes the tanh function starts at the non-saturating area, where the gradient is large enough to initiate the update. The constant $K$ is set to $1/\tanh(\beta_{0})$ so that the CAS is initialized as an identity mapping when the training begins.

To update the parameters, we need to compute gradients of the loss function with respect to $\alpha$, $\beta$, and input value $z_{i}$. Let $\hat{\bm{z}}=({\hat{z}}_{1},{\hat{z}}_{2},...,{\hat{z}}_{d})\in\mathbb{R}^{d}$ be the output of CAS as follows:

Gradients can be derived from the chain rule as follows:

where $\mathcal{L}$ is the loss function and $\frac{\partial\mathcal{L}}{\partial\hat{z}_{i}}$ is obtained from backpropagation. $\frac{\partial p}{\partial z_{i}}$ is the partial differential of the competition indicator $p$ with respect to the input. For $p_{E}$, $p_{L1}$ and $p_{c}$, we ignore the small positive constant value $\epsilon$ and the gradients are derived as follows:

Thus, we obtained all gradients of CAS for gradient descent and backpropagation. Since Eq. (11) and Eq. (12) are just different concatenations of CAS, ReLU, and BN, which are all differentiable with respect to the input and parameters, the gradients can be derived from the chain rule.

We compare our methods to the most popular activation functions on CIFAR-100 imagine classification task [19]. CIFAR-100 is a dataset that has 100 classes, containing 500 training images and 100 test images for each class. We use ResNet-18 [20], GoogLeNet [21], and VGG-13 [22] networks to evaluate our activation functions. A stochastic gradient descent (SGD) optimizer with a momentum of $0.9$ and a weight decay of $5\times 10^{-4}$ is used to train all networks. The learning rate starts at $0.1$ and is divided by $5$ in $50$th, $120$th, and $160$th epochs. The training ends at $200$th epoch.

Table I shows the results of the classification experiment. Most implementations of our method have overall better performance compared to other baseline methods. ${\rm CAReLU_{E}}$ has the highest accuracy in ResNet-18 experiments and ${\rm BN\text{-}CAReLU_{E}}$ has the highest accuracy in GoogLeNet and VGG-13 experiments. When comparing different implementations of our method, ${\rm BN\text{-}CAReLU}$ performs better than ${\rm CAReLU}$ except for the ${\rm CAReLU_{E}}$ in ResNet-18 experiments and ${\rm CAReLU_{c}}$ in GoogLeNet, which proves the previous assumption that BN’s normalization impairs the effectiveness of our method. Energy-based implementations generally perform better than the other two implementations. The reason is that the gradient of ${\rm CAReLU_{E}}$ is a continuous function and it’s smoother than the gradient of ${\rm CAReLU_{L1}}$; ${\rm CAReLU_{c}}$ contains Heaviside step functions, which results in a bumpy loss landscape [23] and thus impairs the performance.

We also evaluate values of $\tanh(\alpha p+\beta)$ in CASs on each layer. In Table II, we show the means and standard deviations of $\tanh(\alpha p+\beta)$ from the first 10 CASs of best models on the test set. Small deviations indicate that CASs on trained models do not generate significantly different values of $\tanh(\alpha p+\beta)$ for different samples. Instead, input values are scaled uniformly and then fine-tuned for each sample according to the input’s competition results. Some degenerate into constant scaling such as CAS#6 of VGG-13 in Table II. Allowing degeneracy into ReLU enables our method to utilize competition results without compromising the performance whenever the original approach is optimal. Histograms of $\alpha p+\beta$ obtained from the best models on the test set are shown in Fig. 1. Despite our initial design being a binary scale factor described in Eq. (4), values of $\alpha p+\beta$ mostly land on non-saturating regions.

We compare our methods to the most popular activation functions on an image super-resolution task on the Berkeley segmentation dataset (BSD300), which contains 200 training images and 100 test images [24]. The network we used for this experiment is an efficient sub-pixel convolutional neural network (ESPCN) featuring a network comprising several convolutional layers and a pixel shuffle layer [25]. Training images are cropped to $256\times 256$ and scaled down to $(256/r)\times(256/r)$ where $r$ is the upscale factor. The ESPCN network upscales the down-scaled images back to $256\times 256$. An Adam optimizer [26] with a learning rate of $1\times 10^{-3}$ is used for training the network in 200 epochs. We run every setting 5 times and show experiment results in Table III.

The data show that ${\rm CAReLU_{E}}$ and ${\rm CAReLU_{L1}}$ surpass other activation functions in PSNR. ${\rm CAReLU_{c}}$ outperforms PReLU, Swish-1, and Swish, but does not perform as well as ${\rm CAReLU_{E}}$ and ${\rm CAReLU_{L1}}$ due to its discontinuity introduced by the Heaviside function.

In this section, we evaluate our methods on the Stanford Natural Language Inference (SNLI) corpus [27]. SNLI corpus is a collection of 570k human-written English sentence pairs labeled for entailment, contradiction, and neutral. A premise and a hypothesis comprise a sentence pair. The model we used for this task comprises an embedding layer, a Long short-term memory (LSTM) [28] encoder, and a sequence of fully connected layers. Hypothesis and premise go through the embedding layer and the encoder independently. We concatenate the encoded hypothesis feature and premise feature, then send them through the fully connected layers for classification. We use the Adam optimizer with a learning rate of $0.001$ to train the parameters for 50 epochs on the SNLI training set. We run every setting 5 times and show experiment results in Table IV.

Data suggest that our methods perform better than other activation functions. Swish and Swish-1 are unable to converge in this task. ${\rm CAReLU_{E}}$ achieve the highest classification accuracy in this experiment. Though mean accuracies of ${\rm CAReLU_{L1}}$ and ${\rm CAReLU_{c}}$ are approximately identical, ${\rm CAReLU_{c}}$ is more robust because it has a smaller standard deviation. All three implementations of our method outperform other baseline activation functions.