A New Accelerated Stochastic Gradient Method with Momentum

Stochastic gradient descent methods with momentum (Sgdm) update parameter $x$ with

where $m_{k}$ is called a momentum term, and Sgdm updates the momentum term with

If $\gamma$ is a constant, then we call it a exponentially decay rate. With (4), we can rewrite the update rule (3) as

which shows that the moving direction here is a weighted average of all gradients, with exponential decay weights. We denote the weighted average of gradients as $v_{k}=\sum_{j=1}^{k}\alpha_{j}\gamma^{k-j}g_{j}(x_{j})$. It is believed that the momentum term can reduce the variance Sutskever et al. (2013). Using a exponential decay rate is common and effective. According our following analyses, a exponential decay rate can limit the variance of $v_{k}$ to a region which determined by the constant $\gamma$.

Main Contribution First, we proposed a novel stochastic gradient descent method with inverse proportional decay rate momentum, which method dynamically adjusts the momentum to match the convergence of the stochastic optimization problem. Besides, our rigorous analyses prove that both the simple SGDM and our novel momentum term can limit the variance to a region $\gamma$ (Theorem 3.4,Theorem 3.6). We list out two main theorems (informally) in the following.

For a differential function $F$ with $L$-Lipschitz gradient and the variance of gradient is limited, Theorem 3.4 implies that the momentum term with exponentially decay weights can limit the variance of $v_{k}$

Theorem 3.6 implies that the momentum term with our inverse proportional decay weights can limit the variance of $v_{k}$

Extensive experiments in Section 4 shows that the robustness of our method extends from linear regression to practical model in deep learning problems.

Momentum method and its use within optimization problems has been studied extensively  Sutskever et al. (2013); Orr (1996); Leen and Orr (1993). The classical momentum (CM) method  Polyak (1964) accumulates a decaying sum of the previous updates of parmeter $x$ into a momentum term $m$ using equation (4) and updates $x$ with (3). CM uses constant hyperparameter $\gamma$ and learning rate $\alpha$. With this method, one can see that the steps tend to accumulate contributions in directions of persistent descent, while directions that oscillate tend to be cancelled, or at least remain small Bottou et al. (2018). Thus, this method can make optimization algorithms move faster along dimensions of low curvature where the update is small and its direction is persistent and slower along turbulent dimensions where the update usually significantly changes its direction  Sutskever et al. (2013). The adaptive momentum method in Leen and Orr (1993) uses decaying learning rate and adaptive $\beta=\max(0,1-\alpha_{0}\mu^{2})$, where $\mu(t)$ is the network input at time $t$.

Our method updates $x_{k}$ with

and it updates the momentum term $m_{K}$ with

where

The hyperparameter $\beta$ is a predefined constant. Thus we can rewrite the updating rules (6),(7), and (8) as

We denote the moving direction of $x_{k}$ as $v_{k}=-\sum_{i=1}^{k}\alpha_{i}g_{i}(x_{i})\left(\frac{i}{k}\right)^{\beta}$.

We use the same stepsize as shown in (adam)

And it is straightforward that $\sum_{i}^{\infty}\alpha_{k}=\infty$.

We first consider a fixed decay factor $\gamma\in(0,1)$, which is used in many simple stochastic gradient descent methods with momentum.

We evaluate the two methods (saying Sgdm, and our method), on two multi-class logistic regression problems using the MNIST dataset without regularization. Logistic regression is suitable for comparing different optimizers for its simple structure and convex objective function. In our experiments, we set the stepsize $\alpha_{t}=\frac{\alpha}{\sqrt{t}}$, where $t$ is the current iteration times. The logistic regression problems is to classify the class label directly on the $28\times 28$ image matrix.

We compare the performance of our method (Lim for short) to optimize the logistic regression problem with that of Sgdm using a minibatch size of 128. According to Fig. 1, we found that our method converges faster than Sgdm.

Multi-layer neural networks are powerful models with non-convex objective functions. We empirically found that our method outperforms Sgdm. In our experiments, the multi-layer neural network models are consistent with that in previous publications, saying a neural network model with two or three fully connected hidden layers with 1000 hidden units each and ReLU activation are used for this experiment with a minibatch size of 128.

We investigate the two optimizers using the standard deterministic cross-entropy objective function without regularization. According to Fig. 2, we found that our method outperforms Sgdm by a large margin.

Deep convolutional neural networks (CNNs) have shown considerable success in practical machine learning tasks, e.g. computer vision tasks and natural language procession tasks. Our CNN architectures has three or two alternating stages of $5\times 5$ convolution filters and $3\times 3$ max pooling with stride of 2, each followed by a fully connected layer of 1000 units with RLeU activations. We pre-process the input image with whitening imply drop out noise to the input layer and fully connected layer. The minibatch size is also set to 128.

We investigate the three optimizers using the standard deterministic cross-entropy objective function. According to Fig. 3, we found that our method converge faster than Adam and Sgdm.