A Method for Enhancing Generalization of Adam by Multiple Integrations

Long Jin¹, Han Nong¹, Liangming Chen¹, Zhenming Su¹

Abstract

The insufficient generalization of adaptive moment estimation (Adam) has hindered its broader application. Recent studies have shown that flat minima in loss landscapes are highly associated with improved generalization. Inspired by the filtering effect of integration operations on high-frequency signals, we propose multiple integral Adam (MIAdam), a novel optimizer that integrates a multiple integral term into Adam. This multiple integral term effectively filters out sharp minima encountered during optimization, guiding the optimizer towards flatter regions and thereby enhancing generalization capability. We provide a theoretical explanation for the improvement in generalization through the diffusion theory framework and analyze the impact of the multiple integral term on the optimizer’s convergence. Experimental results demonstrate that MIAdam not only enhances generalization and robustness against label noise but also maintains the rapid convergence characteristic of Adam, outperforming Adam and its variants in state-of-the-art benchmarks.

Code — https://github.com/LongJin-lab/MIAdam

Introduction

An appropriate optimizer is essential to train a deep neural network (DNN), as it directly affects the training convergence and performance of a model (Yao et al. 2021). The goal of optimizers is usually to minimize (or maximize) a certain objective function, typically a loss function, which measures the gap between the predictions and ground-truth values. As a traditional optimizer, stochastic gradient descent (SGD) is a commonly used optimizer for training DNNs (Deng et al. 2023). However, SGD suffers from certain limitations, such as the need to precisely tune the learning rate, the uniform scaling of gradients in all directions, and the risk of being trapped in saddle points (Johnson et al. 2020; Liu et al. 2021). In order to address these challenges, adaptive learning rate optimizers are developed, offering more nuanced control over learning rates and improved convergence in diverse training scenarios. Among them, adaptive moment estimation (Adam) (Kingma and Ba 2015) is currently one of the most popular adaptive learning optimizers for its rapid convergence and efficient handling of sparse gradients. The combination of first-order and second-order moments in Adam enables the effective incorporation of momentum-based optimization and adaptive learning rate methods, thereby enhancing its overall efficiency and applicability in various neural network training contexts.

Refer to caption — Figure 1: The idea of this work and the filtering effect of integrations on optimizer trajectories. The blue integrated trajectory represents an equivalent path that does not actually exist on the original loss landscape.

Despite being widely used, Adam also exhibits certain limitations, such as the inferior generalization capabilities compared to SGD in some scenarios (Wilson et al. 2017; Luo et al. 2019; Zou et al. 2021). Therefore, several enhanced variants of Adam are developed to alleviate the issue of poor generalization. Switching from Adam to SGD (SWATS) (Keskar and Socher 2017) is designed to start training with the Adam optimizer, then automatically switch to SGD, aiming to improve the model’s generalization performance. However, this method can not maintain the original convergence rate of Adam to some extent. ND-Adam (normalized direction-preserving Adam) (Zhang 2018) meticulously preserves the direction of gradients for each parameter and produces the regularization effect akin to L2 weight decay. Despite this, its ability to enhance generalization is quite limited. AdaBound (Luo et al. 2019) employs dynamic constraints on the learning rate to achieve a smooth and gradual transition from adaptive methods to SGD, which enhances the generalization of a model and reduces the dependence on detailed learning rate adjustments. Nonetheless, a significant drawback of AdaBound is its potential for slow convergence in certain scenarios (Savarese 2019). Overall, these improved optimizers of Adam attempting to enhance the generalization of Adam are unable to simultaneously retain the rapid convergence characteristic of Adam and enhance generalization effectively.

Many studies show theoretically and empirically that the generalization performance of a model is highly correlated with its loss landscape in the parameter space (Hochreiter and Schmidhuber 1997; Chaudhari et al. 2019; Jiang et al. 2020; Petzka et al. 2021; Du et al. 2022). A crucial observation is that flat regions in the loss landscape tend to be associated with good generalization performance, while sharp or narrow regions may lead to overfitting (Mulayoff and Michaeli 2020; Sun et al. 2023). This means that optimizers can effectively improve the generalization of a model by converging to flat minima during the training process, which provides a new perspective to alleviate the issue of the poor generalization of Adam. In order to improve the generalization of a model by finding flat minima in the loss landscape, we propose multiple integral Adam (MIAdam), which is inspired by the effect that the multiple integral term can often serve as filters and noise suppressions in the field of signal processing and control systems (Roberts and Mullis 1987; Jin, Zhang, and Li 2015). MIAdam introduces a multiple integral term to the parameter update formula of Adam, utilizing the filtering effect of multiple integrations to smooth the optimizer’s trajectory. As shown in Fig. 1, if we consider the trajectory as a time-varying input signal, integrating the signal is equivalent to filtering out the sharp minima encountered by the optimizer on the loss landscape, thereby enabling the optimizer to converge to flat minima. More details about the design of the MIAdam optimizer are discussed in Section MIAdam. The main contributions of this paper are summarized as follows.

$\bullet$

To the best of our knowledge, the method of introducing multiple integral term in the optimizer to find flat minima in the loss landscape is proposed for the first time. Furthermore, we propose a new optimizer based on Adam, which is called MIAdam.
$\bullet$

We provide theoretical analyses on MIAdam. Specifically, utilizing the diffusion theory framework in (Xie, Sato, and Sugiyama 2020), we prove that the multiple integral term enables MIAdam to generalize better than Adam under some assumptions. In addition, we also analyze the effect of multiple integral terms on convergence.
$\bullet$

The effectiveness of the proposed method is validated through image classification experiments, text classification experiments, and experiments that inject label noises into datasets. Experimental results demonstrate that MIAdam outperforms the Adam and its state-of-the-art (SOTA) variants on both generalization and robustness against label noises.

Preliminaries

In this section, core concepts about Adam and the related theoretical analyses on the relationship between flat minima and generalization are briefly given to set the stage for the detailed exposition of MIAdam that follows.

Overview of Adam

The training procedure for a DNN can be primarily characterized as an optimization problem, which is defined as follows:

\min_{\bm{\theta}}\frac{1}{\left|S\right|}\sum_{k=1}^{\left|S\right|}\mathscr{L}(\bm{x}_{k},\bm{y}_{k};\bm{\theta}),

(1)

where $\mathscr{L}(\bm{x}_{k},\bm{y}_{k};\bm{\theta})$ represents the loss function; $\bm{\theta}$ denotes the parameter of the model; $\bm{x}_{k}$ and $\bm{y}_{k}$ are the input and its corresponding ground-truth label, respectively; $S$ is a subset of $D$ and $D$ is the training dataset. In the early stages of deep learning development, SGD emerges as a prevailing optimizer, with its parameter update formula expressed as follows:

\theta_{t+1,i}=\theta_{t,i}-\alpha g_{t,i},

(2)

where $\alpha$ represents the learning rate; $\theta_{t,i}$ represents the $i$ -th dimension of the parameter at discrete time $t$ ; $g_{t,i}$ is the gradient with respect to the parameter $\theta_{t,i}$ . The gradient is formally defined as

g_{t,i}=\frac{\partial L(\theta_{t,i})}{\partial\theta_{t,i}},

(3)

where $L(\bm{\theta})=(1/\left|S\right|)\sum\nolimits_{k=1}^{\left|S\right|}\mathscr{L}(\bm{x}_{k},\bm{y}_{k};\bm{\theta})$ . Momentum is a strategy used to expedite convergence of SGD towards minima and escape saddle points on the loss landscape (Qian 1999). It is computed by accumulating previous gradients into the current gradient. The parameter update formula of SGD with momentum (SGDM) is shown as

		$\displaystyle m_{t,i}=\beta m_{t-1,i}+g_{t,i},$			(4a)
		$\displaystyle\theta_{t+1,i}=\theta_{t,i}-\alpha m_{t,i},$			(4b)

where $m_{t,i}$ denotes the momentum at $t$ and $\beta$ is the hyperparameter is used to trade off between the current gradient and the accumulation of historical gradients. Adam refines the momentum formulation in Eq. (4a) and introduces an adaptive learning rate achieved through the computation of the first-order and the second-order moments concerning current gradients. The first-order and second-order moments are calculated by the following expressions:

		$\displaystyle m_{t,i}=\beta_{1}m_{t-1,i}+(1-\beta_{1})g_{t,i},$			(5a)
		$\displaystyle v_{t,i}=\beta_{2}v_{t-1,i}+(1-\beta_{2})g_{t,i}^{2},$			(5b)

where $v_{t,i}$ denotes the second-order moment at $t$ ; the $\beta_{1}$ and $\beta_{2}$ are the exponential decay rates used to adjust the first-order and second-order moments, respectively. Furthermore, the parameter update formula of Adam is expressed as

\displaystyle\vspace{-16pt}\theta_{t+1,i}=\theta_{t,i}-\frac{\alpha\hat{m}_{t,i}}{\sqrt{\hat{v}_{t,i}+\epsilon}},\vspace{-16pt}

(6)

where $\hat{m}_{t,i}=m_{t,i}/(1-\beta_{1}^{t})$ and $\hat{v}_{t,i}=v_{t,i}/(1-\beta_{2}^{t})$ . The hyperparameter $\epsilon$ is assumed as a small value to prevent division by zero in the denominator.

The Relationship between Flat Minima and Generalization

The generalization of DNNs has been extensively explored in recent years. In order to understand the phenomenon of generalization of DNNs, some of the existing research delves into understanding the relationship between loss landscapes and the generalization. A correlation between the flatness of the loss landscape and model generalization is revealed in (Hochreiter and Schmidhuber 1995). Subsequent investigations in (Hochreiter and Schmidhuber 1997) expand on this correlation and provide a method for identifying flat minima. In (Keskar et al. 2017), a definition of the sharpness of a specified point on a loss landscape is given. The study in (Dinh et al. 2017) introduces a reparameterization method and argues that previous sharpness measurements are inadequate for predicting generalization capabilities. Furthermore, it is demonstrated that the generalization capability is influenced by factors such as the batch size, higher-order “smoothness” terms characterized by the Lipschitz constant of the Hessian matrix, the loss function, and the number of parameters (Wang et al. 2018). Based on the above theoretical studies, empirical experiments extensively explore the intrinsic link between the generalization performance of a model and loss landscapes. A consensus emerging from these studies, including (Chaudhari et al. 2019; Jiang et al. 2020; Du et al. 2022; Petzka et al. 2021), is that flat minima usually yield better generalization compared to sharp minima.

MIAdam

In the parameter update formula of Adam, the first-order moment, as defined in Eq. (5a), is reformulated as follows (Kingma and Ba 2015):

\displaystyle m_{t,i}=

\displaystyle(1-\beta_{1})\sum^{t}_{j=0}\beta_{1}^{t-j}g_{j,i}.

(7)

Consequently, the parameter update formula of Adam is rewritten as

\theta_{t+1,i}=\theta_{t,i}-\frac{\alpha(1-\beta_{1})\sum^{t}_{j=0}\beta_{1}^{t-j}g_{j,i}}{(1-\beta_{1}^{t})\sqrt{\hat{v}_{t,i}+\epsilon}}.

(8)

When the learning rate $\alpha$ is sufficiently small, Eq. (8) is approximated in a continuous form as follows:

\displaystyle\mathrm{d}\theta_{\tilde{t},i}

\displaystyle=\mu_{{\tilde{t}},i}\int_{0}^{\tilde{t}}\beta_{1}^{\tilde{t}-\tau}g_{i}(\tau)\mathrm{d}\tau,

(9)

where ${\tilde{t}}$ is the continuous time, $\mathrm{d}\tau$ is equivalent to $\alpha$ and $\mu_{\tilde{t},i}=-(1-\beta_{1})/((1-\beta_{1}^{\tilde{t}})\sqrt{\hat{v}_{{\tilde{t}},i}+\epsilon})$ . It is noteworthy that the integral term appears in Eq. (9). In signal processing, a continuous input signal $x({\tilde{t}})$ that undergoes an integral operation is written as

\displaystyle y(\tilde{t})=\int_{{\tilde{t}}_{0}}^{{\tilde{t}}}x(\tau)\mathrm{d}\tau,

(10)

where $y({\tilde{t}})$ is the integrated signal and the integration range is from ${\tilde{t}}_{0}$ to ${\tilde{t}}$ . Ultimately, the resulting integral signal $y({\tilde{t}})$ contains the cumulative information of the original signal $x({\tilde{t}})$ at different points in time. After the integral operation, the high-frequency components of the signal are filtered out. Inspired by this, the trajectory of the optimizer on the loss landscape can be viewed as the input signal when training a DNN, and the sharp minima are equivalent to the high-frequency components in the signal. Integrating this signal is equivalent to filtering out the sharp minima encountered by an optimizer in the loss landscape, thereby guiding the optimizer toward convergence in flat regions. Therefore, to further achieve the effect of filtering out the sharp minima encountered by the optimizer, multiple integrations, an enhanced version of the integral operation, are introduced into the parameter update formula of Adam. Based on the process involving the integration as depicted in Eq. (10), we obtain the following equation:

		$\displaystyle\mathrm{d}\theta_{{\tilde{t}},\mathcal{i}}=$		(11)
		$\displaystyle\mu_{{\tilde{t}},i}\!\!\overbrace{\int_{0}^{{\tilde{t}}}\!\!\!\kappa^{{\tilde{t}}\!-{\tilde{t}}_{1}}\!\!\!\int_{0}^{{\tilde{t}}_{1}}\!\!\!\!\!\!\cdots\!\!\int_{0}^{{\tilde{t}}_{n\!-\!2}}\!\!\!\kappa^{{\tilde{t}}_{n\!-\!2}\!-{\tilde{t}}_{n\!-\!1}}\!\!\!\int_{0}^{{\tilde{t}}_{n\!-\!1}}}^{\text{$n{\text{-th}}$-order multiple integration}}\!\!\!\beta_{1}^{{\tilde{t}}_{n\!-\!1}\!-\!\tilde{\tau}}\!\!\!\mathcal{g}_{\mathcal{i}}(\tilde{\tau})\mathrm{d}\tilde{\tau}\mathrm{d}{\tilde{t}}_{n\!-\!1}\!\cdots\!\mathrm{d}{\tilde{t}}_{1},$		(11)

where $\kappa$ is multiple integration rate which adjusts the multiple integral term. Then, we perform cumulative operations on the first-order moments in the parameter update formula of Adam to transform the multiple integral term from a continuous form to its corresponding discrete form. Thus, the corresponding parameter update formula of Eq. (LABEL:eq12) is derived as follows:

\left\{\begin{aligned} &m_{t,i}=\beta_{1}m_{t-1,i}+(1-\beta_{1})g_{t,i},\\ &\overline{m}_{t,i}^{(n)}\!\!=\!(1\!-\!\beta_{1})\!\!\!\overbrace{\sum^{t}_{t_{1}=0}\!\!\kappa^{t-t_{1}}\!\!\!\!\sum^{t_{1}}_{t_{2}=0}\!\!\cdots\!\!\!\!\!\!\sum^{t_{n\!-\!2}}_{t_{n\!-\!1}=0}\!\!\!\!\kappa^{t_{n\!-\!2}-t_{n\!-\!1}}\!\!\!\!\sum^{t_{n\!-\!1}}_{t_{n}=0}\!\!\beta_{1}^{{t_{n\!-\!1}}\!-\!t_{n}}}^{\text{$n{\text{-th}}$-order multiple summation}}\!\!\!g_{t_{n},i}\\ &\quad\ \!\ \ =\!\!\sum^{t}_{t_{1}=0}\!\kappa^{t-t_{1}}\!\!\sum^{t_{1}}_{t_{2}=0}\!\!\cdots\!\!\!\!\!\!\sum^{t_{n\!-\!2}}_{t_{n\!-\!1}=0}\!\!\!\!\kappa^{t_{n\!-\!2}-t_{n\!-\!1}}m_{t_{{n\!-\!1},i}},\\ &\theta_{t+1,i}=\theta_{t,i}-\frac{\alpha^{n}\overline{m}_{t,i}^{(n)}}{(1-\beta_{1}^{t})\sqrt{\hat{v}_{t}+\epsilon}},\end{aligned}\right.

(12)

where the superscript ⁽ⁿ⁾ means the $n$ -th-order multiple summation.

Algorithm 1 MIAdam

Given: Learning rate: $\alpha$ ;

exponential decay rates: $\beta_{1}$ , $\beta_{2}$ ;

multiple integration rate: $\kappa$ ;

infinitesimal term: $\epsilon$ ;

the order of the multiple integral item: $n$ ;

switching moment: $\zeta$ .

Initialize: Step time $t\leftarrow 0$ ;

first moment vector $m_{t=0,i}\leftarrow 0$ ;

second moment vector $v_{t,i}\leftarrow 0$ , $\overline{m}_{t=0,i}^{(0)}\leftarrow m_{t=0,i}$ .

1: while stopping criterion is not met do

t\leftarrow t+1

3: using Eq. (3) to get the gradient

g_{t,i}

m_{t,i}\leftarrow\beta_{1}m_{t-1,i}+(1-\beta_{1})g_{t,i}

v_{t,i}\leftarrow\beta_{2}v_{t-1,i}+(1-\beta_{2})g_{t,i}^{2}

6: if

t<\zeta

then

m^{(0)}_{t,i}\leftarrow m_{t,i}

8: for

j=1

n

\overline{m}_{t,i}^{(j)}\leftarrow\kappa\overline{m}_{t-1,i}^{(j)}+\overline{m}_{t,i}^{(j-1)}

10: end for

11:

\alpha_{t}\leftarrow\alpha^{n}

12:

\hat{m}_{t,i}\leftarrow\overline{m}_{t,i}^{(n)}/(1-\beta_{1}^{t})

13: else

14:

\alpha_{t}\leftarrow\alpha

15:

\hat{m}_{t,i}\leftarrow m_{t,i}/(1-\beta_{1}^{t})

16: end if

17:

\hat{v}_{t,i}\leftarrow v_{t,i}/(1-\beta_{2}^{t})

18:

\theta_{t,i}\leftarrow\theta_{t-1,i}-\alpha_{t}\hat{m}_{t,i}/(\sqrt{\hat{v}_{t,i}}+\epsilon)

19: end while

According to the theoretical analyses in Section Generalization and Convergence Analyses and the simulations in Fig. 2, although the multiple integral term helps an optimizer to find flat minima, the optimizer hovers around flat minima and does not converge. Thus, we only use the multiple integral term in the early stages of training, and after that, the optimizer switches to Adam to ensure that the training is convergent eventually. At this point, the multiple integral term is introduced into Adam, and this new optimizer is named MIAdam. The pseudo code for MIAdam is shown in Algorithm 1.

Note that the multiple integration is approximated by the multiple summation in Algorithm 1, which adds only $n$ additional summation operations at each iteration for each dimension of the parameter. Therefore, MIAdam adds very little additional computational overhead compared to Adam. In the following text, we refer to Adam with an additional first-order integration as MIAdam1, and the one with an additional second-order integration as MIAdam2, and so on.

Generalization and Convergence Analyses

In this section, we present the theoretical analyses of the generalization and convergence associated with the addition of the multiple integral term to Adam, which does not involve the switching of optimizers. These analyses provide a theoretical foundation for our proposed optimizer.

Generalization Analyses

In this subsection, the diffusion theory framework is utilized to rigorously demonstrate that the incorporation of the multiple integral term enhances the generalization capabilities of the model. Specifically, generalization is quantitatively assessed by comparing the mean escape time, represented as $\phi$ , which indicates an optimizer’s ability to escape from sharp minima. In the following analyses, we begin by delineating three fundamental assumptions that are crucial for the application of the diffusion theory framework (Xie, Sato, and Sugiyama 2020).

Assumption 1.

The loss function around the critical point $\bm{p}$ is approximately written as

\displaystyle L(\bm{\theta})=L(\bm{p})+\frac{1}{2}(\bm{\theta}-\bm{p})^{\top}H(\bm{p})(\bm{\theta}-\bm{p}),

(13)

where the superscript ^⊤ means the transpose of a vector.

Assumption 2.

(Quasi-equilibrium approximation). The system is in quasi-equilibrium near minima.

Assumption 3.

(Low-temperature approximation). The system is under low temperature (small gradient noise).

Consequently, following the theoretical analyses in (Xie, Sato, and Sugiyama 2020; Xie et al. 2022), we can further deduce Theorem 1. The detailed proof is given in the Appendix.

Theorem 1.

Suppose that Assumption 1, Assumption 2, and Assumption 3 hold while saddle point $\bm{u}$ is the exit from sharp minimum $\bm{a}$ . Then the mean escape time of MIAdam1 from sharp minimum $\bm{a}$ to flat minimum $\bm{b}$ through saddle point $\bm{u}$ before the switch is

		$\displaystyle\phi_{\mathrm{MIAdam1}}=$		(14)
		$\displaystyle\pi\left[\sqrt{1+\frac{4\alpha\sqrt{\mathcal{b}\left\|H_{{\bm{u}}{\bm{e}}}\right\|}}{\tilde{t}(1-\beta_{1})}}+1\right]\frac{\left\|\operatorname{det}\left(H_{\bm{a}}^{-1}H_{\bm{u}}\right)\right\|^{\frac{1}{4}}}{\left\|H_{{\bm{u}}{\bm{e}}}\right\|}$
		$\displaystyle\exp\left[\frac{2\sqrt{\mathcal{b}}\Delta L}{\tilde{t}\alpha}\left(\frac{\varrho}{\sqrt{H_{{\bm{a}}{\bm{e}}}}}+\frac{(1-\varrho)}{\sqrt{\left\|H_{{\bm{u}}{\bm{e}}}\right\|}}\right)\right],$

where subscript _e denotes the escape direction; $\varrho$ is the path-dependent parameter; $\mathcal{b}=|S|$ indicates the batch size; $\Delta L=L(\bm{u})-L(\bm{a})$ ; $H$ represents the Hessian matrix.

Comparing the mean escape time $\phi_{\mathrm{MIAdam1}}$ obtained from Theorem 1 with that of Adam’s in (Xie et al. 2022),

	$\displaystyle\phi_{\mathrm{Adam}}=$	$\displaystyle\pi\left[\sqrt{1+\frac{4\alpha\sqrt{\mathcal{b}\left\|H_{\bm{u}\bm{e}}\right\|}}{(1-\beta_{1})}}+1\right]\frac{\left\|\operatorname{det}\left(H_{\bm{a}}^{-1}H_{\bm{u}}\right)\right\|^{\frac{1}{4}}}{\left\|H_{{\bm{u}}{\bm{e}}}\right\|}$		(15)
		$\displaystyle\exp\left[\frac{2\sqrt{\mathcal{b}}\Delta L}{\alpha}\left(\frac{\varrho}{\sqrt{H_{{\bm{a}}{\bm{e}}}}}+\frac{(1-\varrho)}{\sqrt{\left\|H_{{\bm{u}}{\bm{e}}}\right\|}}\right)\right],$		(15)

when $\tilde{t}>1$ , it is found that $\phi_{\mathrm{MIAdam1}}$ is smaller than $\phi_{\mathrm{Adam}}$ , indicating that Adam introduces an additional first-order integration which is more likely to escape from sharp minima and consequently converge to flat minima, thereby improving the generalization.

Convergence Analyses

In order to verify the effect of the multiple integral term on the convergence of the optimizer, we follow the analytical framework of Adam which is also used in this subsection. Concretely, the regret bound $R({\hat{t}})$ is utilized to evaluate the convergence of the algorithm and is defined as follows:

\displaystyle R({\hat{t}})=\sum_{t=1}^{\hat{t}}f_{t}(\bm{\theta}_{t})-\min_{\bm{\theta}}\sum_{t=1}^{\hat{t}}f_{t}(\bm{\theta}),

(16)

where $f_{t}(\cdot)$ is a convex loss function.

Theorem 2.

Assume that the convex function $f_{t}$ has bounded gradients, $\left\|\nabla f_{t}(\bm{\theta})\right\|_{2}\leq\mathsf{g},\left\|\nabla f_{t}(\bm{\theta})\right\|_{\infty}\leq$ $\mathsf{g}_{\infty}$ for all $\bm{\theta}\in\mathbb{R}^{d}$ and distance between any $\bm{\theta}_{t}$ is guaranteed to be bounded, $\left\|\bm{\theta}_{n}-\bm{\theta}_{m}\right\|_{2}\leq\mathsf{d}$ , $\left\|\bm{\theta}_{m}-\bm{\theta}_{n}\right\|_{\infty}\leq\mathsf{d}_{\infty}$ for any $m,n\in\{1,\ldots,{\hat{t}}\}$ , and $\beta_{1},\beta_{2}\in[0,1)$ satisfy $\beta_{1}^{2}/\sqrt{\beta_{2}}<1$ . Let $\alpha_{t}=\alpha/t^{h}$ and $\beta_{1,t}=\beta_{1}\lambda^{t-1},\kappa\in(0,1],\lambda\in(0,1)$ . For the convex problem, the $R({\hat{t}})$ of MIAdam1 before the switch satisfies

\displaystyle\lim_{{\hat{t}}\rightarrow\infty}\frac{R({\hat{t}})}{{\hat{t}}}\neq 0.

(17)

The detailed proof of Theorem 2 is thoroughly presented in the Appendix. From Theorem 2, it is evident that merely adding an extra first-order integration to the parameter updating formula of Adam leads to the non-convergence of the optimizer. Although it is non-convergent, it effectively escapes sharp minima and hovers around flat minima in the loss landscape. This observation is corroborated by the simulation results shown in Figs. 2(f)-(h). As a result, the MIAdam’s algorithm is structured to switch to Adam after a certain number of epochs to guarantee convergence.

Simulations and Experiments

In this section, we conduct the simulations on 2-parameter loss landscapes to illustrate the efficiency of MIAdam to escape from sharp minima. Furthermore, extensive empirical experiments are conducted to demonstrate that MIAdam outperforms Adam in terms of generalization and robustness against label noises.

Optimizer	ResNet18				ResNet50
	CIFAR-10(%)	Time	CIFAR-100(%)	Time	CIFAR-10(%)	Time	CIFAR-100(%)	Time
Adam	$93.89_{\pm 0.18}$	47m	$73.17_{\pm 0.26}$	47m	$93.69_{\pm 0.28}$	2h 45m	$75.24_{\pm 0.62}$	2h 47m
NAdam	$93.92_{\pm 0.11}$	46m	$73.08_{\pm 0.31}$	46m	$94.03_{\pm 0.07}$	2h 57m	$74.95_{\pm 0.08}$	2h 58m
AdamW	$93.74_{\pm 0.07}$	48m	$72.11_{\pm 0.23}$	47m	$93.80_{\pm 0.11}$	2h 45m	$74.40_{\pm 0.61}$	2h 48m
ND-Adam	$93.69_{\pm 0.08}$	44m	$72.26_{\pm 0.34}$	48m	$93.81_{\pm 0.18}$	2h 39m	$74.18_{\pm 0.35}$	2h 57m
Adamax	$93.68_{\pm 0.25}$	1h 28m	$74.27_{\pm 0.19}$	1h 27m	$94.10_{\pm 0.19}$	2h 48m	$76.30_{\pm 0.10}$	2h 52m
AdaBound	$92.93_{\pm 0.11}$	48m	$72.51_{\pm 0.23}$	49m	$93.14_{\pm 0.29}$	3h 1m	$73.87_{\pm 0.27}$	3h 2m
SWATS	$92.73_{\pm 0.10}$	1h 6m	$72.76_{\pm 0.27}$	54m	$92.38_{\pm 1.38}$	2h 51m	$72.83_{\pm 2.10}$	2h 50m
Adai	$93.86_{\pm 0.15}$	53m	$74.82_{\pm 0.09}$	53m	$93.95_{\pm 0.06}$	3h 32m	$76.07_{\pm 0.36}$	3h 18m
MIAdam1	$94.20_{\pm 0.12}$ *	47m	$75.03_{\pm 0.05}$ *	47m	$94.17_{\pm 0.10}$	2h 49m	$76.96_{\pm 0.31}$ *	2h 47m
MIAdam2	$94.03_{\pm 0.12}$	47m	$74.41_{\pm 0.38}$	46m	$\bm{94.28_{\pm 0.15}}$ *	2h 47m	$76.63_{\pm 0.15}$	2h 46m
MIAdam3	$93.96_{\pm 0.17}$	47m	$74.53_{\pm 0.22}$	47m	$\bm{94.10_{\pm 0.10}}$	2h 48m	$76.51_{\pm 0.02}$	2h 46m
	DenseNet121				PyramidNet110
	CIFAR-10(%)	Time	CIFAR-100(%)	Time	CIFAR-10(%)	Time	CIFAR-100(%)	Time
Adam	$94.11_{\pm 0.01}$	3h 30m	$73.88_{\pm 0.34}$	3h 32m	$93.45_{\pm 0.10}$	2h 46m	$72.20_{\pm 0.09}$	2h 50m
NAdam	$94.39_{\pm 0.30}$	4h 23m	$65.71_{\pm 0.34}$	4h 24m	$93.33_{\pm 0.20}$	3h 34m	$71.26_{\pm 0.44}$	3h 32m
AdamW	$94.16_{\pm 0.11}$	3h 25m	$75.07_{\pm 0.15}$	3h 30m	$93.25_{\pm 0.08}$	2h 48m	$70.41_{\pm 0.69}$	2h 43m
ND-Adam	$94.11_{\pm 0.01}$	3h 23m	$74.59_{\pm 0.20}$	3h 34m	$93.00_{\pm 0.15}$	2h 58m	$70.72_{\pm 0.30}$	3h 11m
Adamax	$90.97_{\pm 0.15}$	3h 40m	$63.48_{\pm 0.08}$	3h 47m	$92.46_{\pm 0.24}$	4h 51m	$69.43_{\pm 0.28}$	5h 16m
AdaBound	$93.14_{\pm 0.10}$	4h 0m	$73.88_{\pm 0.35}$	4h 2m	$92.14_{\pm 0.22}$	3h 32m	$68.97_{\pm 0.39}$	3h 27m
SWATS	$93.64_{\pm 0.94}$	3h 35m	$75.62_{\pm 3.10}$	3h 45m	$89.42_{\pm 4.13}$	3h 35m	$49.82_{\pm 25.13}$	2h 29m
Adai	$94.45_{\pm 0.21}$	4h 28m	$76.77_{\pm 0.31}$	4h 42m	$93.50_{\pm 0.10}$	4h 18m	$71.94_{\pm 0.31}$	4h 5m
MIAdam1	$94.75_{\pm 0.10}$ *	3h 29m	$77.02_{\pm 0.10}$ *	3h 26m	$93.65_{\pm 0.08}$ *	2h 59m	$72.51_{\pm 0.24}$ *	2h 56m
MIAdam2	$94.43_{\pm 0.12}$	3h 29m	$76.21_{\pm 0.33}$	3h 32m	$93.02_{\pm 0.10}$	2h 52m	$71.96_{\pm 0.59}$	2h 52m
MIAdam3	$94.35_{\pm 0.03}$	3h 30m	$76.54_{\pm 0.28}$	3h 30m	$93.07_{\pm 0.25}$	2h 54m	$71.34_{\pm 0.17}$	2h 53m

Table 1: Top-1 test accuracy (mean

\pm

std) on CIFAR-10 and CIFAR-100.

Simulations

This subsection mainly includes simulations demonstrating that MIAdam is easier to escape from a sharp minima compared to Adam and exploring the impact of learning rate on the optimization process. The first simulation is conducted on an elaborate 2-parameter loss landscape (Yang 2020) with one flat minima surrounded by two sharp minima, whose contour map is displayed in Fig. 2(a). On this loss landscape, the learning rates of Adam and MIAdam1 are respectively set to 0.05, 0.1, and 0.15, and the simulation results for their corresponding optimization trajectories are shown in Figs. 2(b)-(d). It is clear that MIAdam1 tends to escape from sharp minima and converge to flat minima compared to Adam on the 2-parameter loss landscape. Moreover, as the learning rate increases, MIAdam1 is able to converge to the flat minima. In contrast, Adam always shows poor convergence on this loss landscape and can not converge well to the flat minima or sharp minima. Therefore, our proposed method is effective in finding flat minima and can not be simply replaced by increasing the learning rate of Adam.

The second 2-parameter loss landscape used for simulations is depicted in Fig. 2(e), which contains a large number of sharp minima and flat minima. On this loss landscape, the optimization trajectories of MIAdam1, MIAdam2, MIAdam3, and Adam are compared in Fig. 2(f)-(h), respectively. Simulation results indicate that MIAdam1, MIAdam2, and MIAdam3 tend to converge toward flat minima, while the Adam optimizer tends to converge to the nearest sharp minima. It’s worth noting that MIAdam3 exhibits more intense oscillations near the flat region compared to the MIAdam2. This suggests that increasing the order of multiple integration does not always lead to improved outcomes. Different starting points influence the trajectory of the optimizer. Therefore, to make the simulation results more convincing, we conduct additional simulations using 2,500 different starting coordinate points and calculate the sum of the absolute values of the eigenvalues of the Hessian matrix of the different optimizers for the final convergence points at different starting coordinate points and compare them. Due to space constraints, the simulation results are presented in the Appendix.

Experiments

The effectiveness of MIAdam is evaluated in this subsection through extensive empirical experiments. Initially, we conduct image classification experiments with various neural network architectures on CIFAR¹¹1http://www.cs.toronto.edu/ kriz/cifar.html and ImageNet-1k²²2https://www.image-net.org/, compared against widely-used adaptive learning rate optimizers, including Adam and its SOTA variants. Additionally, we utilize the fast computation method of Hessian information of loss landscapes provided in (Yao et al. 2020) for further comparative analyses. Subsequently, the effectiveness of the proposed MIAdam optimizer for text classification tasks is tested using the BERT and RoBERTa models across four distinct datasets (Lin et al. 2021). Finally, to validate the robustness against label noises of MIAdam, we perform image classification experiments on datasets injected with label noises. The results of MIAdam exceeding Adam are all bold, and the optimal experimental results are all marked by asterisks. Because of space constraints, the detailed experimental settings for all experiments are included in the Appendix.

Image Classification Experiments

To enhance the conviction of our experimental results, we employ four different neural network architectures for image classification tasks on the CIFAR-10 and CIFAR-100 datasets: ResNet18 (He et al. 2016), ResNet50 (He et al. 2016), DenseNet121 (Huang et al. 2017), and PyramidNet110 (Han, Kim, and Kim 2017). For experiments on large-scale image datasets, we utilize the AlexNet (Krizhevsky, Sutskever, and Hinton 2012), ResNet18, and DenseNet121 architectures for both training and testing on the ImageNet-1k. The classification performance of MIAdam is compared with optimizers such as Adam, NAdam (Dozat 2016), AdamW (Loshchilov and Hutter 2017), ND-Adam, Adamax (Kingma and Ba 2015), AdaBound, SWATS, and Adai (Xie et al. 2022). Detailed hyperparameters and experimental settings are presented in the Appendix. As observed from Table 1 and Table 2, MIAdam maintains a training time comparable to Adam while obtaining much better performance than Adam. To provide a comparison of the flatness in the final convergence regions, we compute top Hessian eigenvalues, Hessian traces, and full Hessian eigenvalue densities for loss landscapes of Adam, MIAdam1, MIAdam2, and MIAdam3 using DenseNet121 on the CIFAR-100 dataset in Fig. 3. Fig. 3 suggests that the multiple integral term is helpful in finding flatter minima in a specific neural network training task.

Optimizer	AlexNet(%)	ResNet18(%)	DenseNet121(%)
Adam	$46.48$	$67.19$	$71.48$
MIAdam1	$52.34$ *	$72.27$ *	$75.39$ *
MIAdam2	$49.61$	$70.70$	$74.22$
MIAdam3	$43.36$	$70.31$	$\bm{74.60}$

Table 2: Top-1 test accuracies (mean

\pm

std) on ImageNet-1k

Dataset	Optimizer	BERT(%)	RoBERTa(%)
R8	Adam	$98.15_{\pm 0.02}$	$98.36_{\pm 0.05}$
	MIAdam1	$\bm{98.18_{\pm 0.03}}$ *	$\bm{98.45_{\pm 0.05}}$ *
	MIAdam2	$98.11_{\pm 0.10}$	$98.29_{\pm 0.10}$
	MIAdam3	$98.04_{\pm 0.09}$	$98.29_{\pm 0.06}$
R52	Adam	$96.36_{\pm 0.21}$	$96.21_{\pm 0.13}$
	MIAdam1	$96.42_{\pm 0.23}$	$96.48_{\pm 0.01}$
	MIAdam2	$\bm{96.57_{\pm 0.07}}$	$\bm{96.46_{\pm 0.23}}$
	MIAdam3	$\bm{96.65_{\pm 0.27}}$ *	$\bm{96.65_{\pm 0.07}}$ *
MR	Adam	$86.03_{\pm 0.30}$	$87.72_{\pm 0.09}$
	MIAdam1	$86.51_{\pm 0.09}$ *	$89.73_{\pm 0.17}$ *
	MIAdam2	$86.45_{\pm 0.19}$	$89.54_{\pm 0.19}$
	MIAdam3	$\bm{86.34_{\pm 0.09}}$	$89.54_{\pm 0.18}$

Table 3: Test accuracies (mean

\pm

std) on text classification experiments

Text Classification Experiments

We conduct text classification experiments by fine-tuning the pre-trained models, BERT and RoBERTa models, on three widely-used text datasets R8, R52, and Movie Review (MR). Each optimizer is run three times on each dataset using different network structures, with the mean and standard deviation of the test accuracy reported in Table 3. The experimental results indicate that MIAdam significantly outperforms Adam in text classification tasks.

Robustness Against Label Noises

In this subsection, we investigate the capacity of MIAdam to withstand label noises in the training dataset, thereby validating its robustness against label noises. The ResNet18 network is trained by using Adam and MIAdam on the corrupted version of the CIAFR10 dataset, where some of its training labels are randomly flipped while the inputs are kept clean. The noise levels are 20%, 40%, 60%, and 80%. On each noise level, each optimizer is run only once. The remaining experimental settings are consistent with those used in the previous image classification experiments. As indicated in Table 4, MIAdam consistently achieves the highest test accuracy across all noise levels, underscoring MIAdam’s superior robustness against label noises.

Optimizer	Noise rate(%)
Optimizer	20	40	60	80
Adam	$88.24$	$84.90$	$79.61$	$66.39$
ND-Adam	$87.52$	$84.10$	$78.34$	$63.51$
AdaBound	$86.51$	$82.46$	$76.58$	$57.86$
SWATS	$89.43$	$85.47$	$80.30$	$53.50$
Adai	$86.09$	$81.92$	$75.72$	$58.60$
MIAdam1	$90.32$ *	$87.67$ *	$82.02$ *	$67.68$ *
MIAdam2	$89.13$	$85.03$	$\bm{79.84}$	$64.96$
MIAdam3	$88.71$	$85.87$	$79.40$	$64.27$

Table 4: Top-1 test accuracy on CIFAR-10 under label noises.

Conclusion

In this paper, we have proposed MIAdam, a new adaptive learning rate optimizer algorithm with a multiple integral term added to Adam. MIAdam smoothes the optimization trajectory through the filtering effect of the multiple integral term, enabling it to escape sharp local minima during training and converge towards flat minima, thereby alleviating the problem of poor generalization of Adam and improving the robustness against label noises while retaining the fast convergence of Adam. Utilizing the diffusion theory framework, we have provided the proof that incorporating the multiple integral term enhances the capability of the optimizer to escape sharp minima and converge to flatter minima, thus improving the generalization of the models. We have analyzed the convergence of MIAdam and provided a guarantee of convergence. The simulations have demonstrated that MIAdam is capable of finding flatter minima compared to Adam. For empirical analyses, We have conducted image classification experiments, text classification experiments, and experiments that inject label noises into datasets. The experimental results show that MIAdam has much better generalization and robustness against label noises than Adam. Future work will focus on introducing multiple integral terms into other optimizers.

Acknowledgments

This work was supported in part by the National Natural Science Foundation of China under Grant 62476115 and Grant 62176109, in part by the Fundamental Research Funds for the Central Universities under Grant lzujbky2023-ct05 and Grant Izuibky-2023-ey07, in part by the China Computer Federation (CCF)-Baidu Open Fund under Grant 202306, and in part by the Supercomputing Center of Lanzhou University.

References

Chaudhari et al. (2019) Chaudhari, P.; et al. 2019. Entropy-SGD: Biasing gradient descent into wide valleys. Journal of Statistical Mechanics: Theory and Experiment, 2019(12): 124018.
Deng et al. (2023) Deng, X.; Sun, T.; Li, S.; and Li, D. 2023. Stability-based generalization analysis of the asynchronous decentralized SGD. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 37, 7340–7348.
Dinh et al. (2017) Dinh, L.; Pascanu, R.; Bengio, S.; and Bengio, Y. 2017. Sharp minima can generalize for deep nets. In International Conference on Machine Learning, 1019–1028.
Dozat (2016) Dozat, T. 2016. Incorporating Nesterov momentum into Adam. In International Conference on Machine Learning.
Du et al. (2022) Du, J.; Zhou, D.; Feng, J.; Tan, V.; and Zhou, J. T. 2022. Sharpness-aware training for free. Advances in Neural Information Processing Systems, 23439–23451.
Han, Kim, and Kim (2017) Han, D.; Kim, J.; and Kim, J. 2017. Deep pyramidal residual networks. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 5927–5935.
He et al. (2016) He, K.; Zhang, X.; Ren, S.; and Sun, J. 2016. Deep residual learning for image recognition. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 770–778.
Hochreiter and Schmidhuber (1995) Hochreiter, S.; and Schmidhuber, J. 1995. Simplifying neural nets by discovering flat minima. Advances in Neural Information Processing Systems, 7: 529–536.
Hochreiter and Schmidhuber (1997) Hochreiter, S.; and Schmidhuber, J. 1997. Flat minima. Neural Computation, 9(1): 1–42.
Huang et al. (2017) Huang, G.; Liu, Z.; van der Maaten, L.; and Weinberger, K. Q. 2017. Densely Connected Convolutional Networks. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition.
Jiang et al. (2020) Jiang, Y.; Neyshabur, B.; Mobahi, H.; Krishnan, D.; and Bengio, S. 2020. Fantastic Generalization Measures and Where to Find Them. In International Conference on Learning Representations.
Jin, Zhang, and Li (2015) Jin, L.; Zhang, Y.; and Li, S. 2015. Integration-enhanced Zhang neural network for real-time-varying matrix inversion in the presence of various kinds of noises. IEEE Transactions on Neural Networks and Learning Systems, 27(12): 2615–2627.
Johnson et al. (2020) Johnson, T.; Agrawal, P.; Gu, H.; and Guestrin, C. 2020. AdaScale SGD: A user-friendly algorithm for distributed training. In International Conference on Machine Learning, 4911–4920.
Kalinay and Percus (2012) Kalinay, P.; and Percus, J. K. 2012. Phase space reduction of the one-dimensional Fokker-Planck (Kramers) equation. Journal of Statistical Physics, 148(6): 1135–1155.
Keskar et al. (2017) Keskar, N. S.; Mudigere, D.; Nocedal, J.; Smelyanskiy, M.; and Tang, P. T. P. 2017. On large-batch training for deep learning: Generalization gap and sharp minima. In International Conference on Learning Representations.
Keskar and Socher (2017) Keskar, N. S.; and Socher, R. 2017. Improving generalization performance by switching from Adam to SGD. arXiv preprint arXiv:1712.07628.
Kingma and Ba (2015) Kingma, D. P.; and Ba, J. 2015. Adam: A method for stochastic optimization. In International Conference on Learning Representations.
Krizhevsky, Sutskever, and Hinton (2012) Krizhevsky, A.; Sutskever, I.; and Hinton, G. E. 2012. ImageNet classification with deep convolutional neural networks. Advances in Neural Information Processing Systems, 1097–1105.
Lin et al. (2021) Lin, Y.; Meng, Y.; Sun, X.; Han, Q.; Kuang, K.; Li, J.; and Wu, F. 2021. BERTGCN: Transductive text classification by combining GCN and BERT. arXiv preprint arXiv:2105.05727.
Liu et al. (2021) Liu, Z.; Li, B.; Simon, J. B.; and Ueda, M. 2021. SGD can converge to local maxima. In International Conference on Learning Representations.
Loshchilov and Hutter (2017) Loshchilov, I.; and Hutter, F. 2017. Decoupled weight decay regularization. arXiv preprint arXiv:1711.05101.
Luo et al. (2019) Luo, L.; Xiong, Y.; Liu, Y.; and Sun, X. 2019. Adaptive gradient methods with dynamic bound of learning rate. arXiv preprint arXiv:1902.09843.
Mulayoff and Michaeli (2020) Mulayoff, R.; and Michaeli, T. 2020. Unique properties of flat minima in deep networks. In International Conference on Machine Learning, 7108–7118.
Petzka et al. (2021) Petzka, H.; Kamp, M.; Adilova, L.; Sminchisescu, C.; and Boley, M. 2021. Relative flatness and generalization. Advances in Neural Information Processing Systems, 18420–18432.
Qian (1999) Qian, N. 1999. On the momentum term in gradient descent learning algorithms. Neural Networks, 12(1): 145–151.
Roberts and Mullis (1987) Roberts, R. A.; and Mullis, C. T. 1987. Digital signal processing. Addison-Wesley Longman Publishing Co., Inc.
Savarese (2019) Savarese, P. 2019. On the convergence of AdaBound and its connection to SGD. arXiv preprint arXiv:1908.04457.
Sun et al. (2023) Sun, Y.; Shen, L.; Chen, S.; Ding, L.; and Tao, D. 2023. Dynamic Regularized Sharpness Aware Minimization in Federated Learning: Approaching Global Consistency and Smooth Landscape. In International Conference on Machine Learning.
Wang et al. (2018) Wang, H.; Keskar, N. S.; Xiong, C.; and Socher, R. 2018. Identifying generalization properties in neural networks. arXiv preprint arXiv:1809.07402.
Wilson et al. (2017) Wilson, A. C.; Roelofs, R.; Stern, M.; Srebro, N.; and Recht, B. 2017. The marginal value of adaptive gradient methods in machine learning. Advances in Neural Information Processing Systems, 4148–4158.
Xie, Sato, and Sugiyama (2020) Xie, Z.; Sato, I.; and Sugiyama, M. 2020. A diffusion theory for deep learning dynamics: Stochastic gradient descent exponentially favors flat minima. In International Conference on Machine Learning.
Xie et al. (2022) Xie, Z.; Wang, X.; Zhang, H.; Sato, I.; and Sugiyama, M. 2022. Adaptive inertia: Disentangling the effects of adaptive learning rate and momentum. In International Conference on Machine Learning, 24430–24459.
Yang (2020) Yang, X. 2020. Stochastic gradient variance reduction by solving a filtering problem. arXiv preprint arXiv:2012.12418.
Yao et al. (2020) Yao, Z.; Gholami, A.; Keutzer, K.; and Mahoney, M. W. 2020. Pyhessian: Neural networks through the lens of the hessian. In 2020 IEEE International Conference on Big Data (Big Data), 581–590.
Yao et al. (2021) Yao, Z.; Gholami, A.; Shen, S.; Mustafa, M.; Keutzer, K.; and Mahoney, M. 2021. Adahessian: An adaptive second order optimizer for machine learning. In proceedings of the AAAI conference on artificial intelligence, volume 35, 10665–10673.
Zhang (2018) Zhang, Z. 2018. Improved Adam optimizer for deep neural networks. In 2018 IEEE/ACM 26th International Symposium on Quality of Service (IWQoS), 1–2.
Zou et al. (2021) Zou, D.; Cao, Y.; Li, Y.; and Gu, Q. 2021. Understanding the generalization of Adam in learning neural networks with proper regularization. arXiv preprint arXiv:2108.11371.

Appendix A Appendix

Appendix B Simulation and Experiment Settings

All experimental procedures are implemented using Python 3.6 and PyTorch 1.8.3. Computational tasks are executed on a system running Ubuntu 16.04 TLS, equipped with an array of 10 NVIDIA RTX 2080Ti GPUs.

Simulation Settings

The simulation settings are illustrated in this paragraph. All simulations employ the CosineAnnealingLR as the learning rate adjustment strategy, with a total of 1500 training steps. These hyperparameters are set identically for all optimizers: $\epsilon=1\mathrm{e}-8$ , $\beta_{1}=0.9$ , $\beta_{2}=0.999$ , $\alpha=1\mathrm{e}-3$ , and $weight\_decay=5\mathrm{e}-5$ . For the simulations on the loss landscape Fig. 2(a), the hyperparameters $\kappa$ and $\zeta$ for MIAdam are set to 0.885 and 1400, respectively. The same settings are applied for the simulations on the loss landscape Fig. 2(e), with the only difference being the learning rate $\alpha=0.005$ .

Experiment Settings

Image Classification Experiments: For MIAdam, we adopt the same hyperparameters as those used for Adam to ensure a consistent comparison basis. Additionally, the hyperparameters $\kappa$ and $\zeta$ are identified with their optimal values through grid search within the range $[0.01,0.02,...,0.99]$ and $[1,2,...,150]$ . After grid search, the optimal values of $\kappa$ and $\zeta$ are 0.98 and 20, respectively. For each optimizer, we calculate the mean and standard deviation of the top-1 test accuracy based on three individual runs while computing the average training time for three runs. For image classification experiments on the CIFAR datasets, the hyperparameter configurations for various optimizers are provided in Table 5.

Table 5: Hyperparameter settings for optimizers of image classification experiments.

	Epoch $=150$ ; batch size $=128$ ; milestone $=[50,75]$
Adam	$\alpha=1\mathrm{e}-3,\epsilon=1\mathrm{e}-8$ , $\beta_{1}=0.9$ , $\beta_{2}=0.999$ , $weight\_decay=5\mathrm{e}-5$
NAdam	$\alpha=1\mathrm{e}-3,\epsilon=1\mathrm{e}-8$ , $\beta_{1}=0.9$ , $\beta_{2}=0.999$ , $weight\_decay=5\mathrm{e}-5$ , $\gamma=1\mathrm{e}-3$
ND-Adam	$\alpha=1\mathrm{e}-3,\epsilon=1\mathrm{e}-8$ , $\beta_{1}=0.9$ , $\beta_{2}=0.999$ , $weight\_decay=5\mathrm{e}-5$
Adamax	$\alpha=2\mathrm{e}-3,\epsilon=1\mathrm{e}-8$ , $\beta_{1}=0.9$ , $\beta_{2}=0.999$ , $weight\_decay=5\mathrm{e}-5$
AdaBound	$\alpha=1\mathrm{e}-3,\epsilon=1\mathrm{e}-8$ , $\beta_{1}=0.9$ , $\beta_{2}=0.999$ , $weight\_decay=5\mathrm{e}-5$
SWATS	$\alpha=1\mathrm{e}-3,\epsilon=1\mathrm{e}-8$ , $\beta_{1}=0.9$ , $\beta_{2}=0.999$ , $weight\_decay=5\mathrm{e}-5$
Adai	$\alpha=1\mathrm{e}-3,\epsilon=1\mathrm{e}-8$ , $\beta_{1}=0.9$ , $\beta_{2}=0.999$ , $weight\_decay=5\mathrm{e}-5$
MIAdam	$\alpha=1\mathrm{e}-3,\epsilon=1\mathrm{e}-8$ , $\beta_{1}=0.9$ , $\beta_{2}=0.999$ , $weight\_decay=5\mathrm{e}-5$ , $\kappa=0.98$ , $\zeta=20$

For the experiments conducted on the ImageNet-1k dataset, all settings remain consistent with those used for the CIFAR dataset experiments, except for the total number of training epochs, which is set to 90.

Text Classification Experiments: We fine-tune the BERT and RoBERTa models using a learning rate of 0.0001 and a batch size of 128 for 30 epochs. For the two optimizers used in training, Adam and MIAdam, both employ milestones as the learning rate adjustment strategy, with the same hyperparameter settings as follows: $\alpha=1\mathrm{e}-5$ , $\epsilon=1\mathrm{e}-8$ , $\beta_{1}=0.9$ , $\beta_{2}=0.999$ , $\alpha=1\mathrm{e}-3$ , and $weight\_decay=5\mathrm{e}-5$ . Additionally, for MIAdam, the extra hyperparameters $\kappa$ and $\zeta$ are set to 0.98 and 20, respectively.

Experiments on Datasets Injected with Label Noises: The extra hyperparameter of MIAdam $\zeta$ is set to 40. Apart from the injection of label noises on the CIFAR-10 dataset, the experimental settings are identical to those of the image classification experiments.

Appendix C Supplementary Experiments and Simulations

Simulations

On the second loss landscape in Fig. 2(e), different starting points influence the trajectory of the optimizer. Therefore, to make the simulation results more convincing, we uniformly select 2,500 starting coordinate points on this loss landscape and conduct 2,500 rounds of simulations in the region where $\theta_{1}\in[-2,3]$ and $\theta_{2}\in[-2,3]$ . Furthermore, we calculate the sum of the absolute values of the eigenvalues of the Hessian matrix of the different optimizers for the final convergence points at different starting coordinate points and compare them. As can be seen in Fig. 4, the sum of the absolute values of the eigenvalues at the location of the final convergence on the second loss landscape is generally smaller for the MIAdam2 and MIAdam3 compared to Adam, which suggests that the multiple integral term is indeed helpful in finding flat minima.

Experiments

Fig. 5 provides training and testing comparisons of Adam, MIAdam1, MIAdam2, and MIAdam3 on CIFAR-100 using DenseNet121. Fig. 5 shows that the introduction of the multiple integral term leads to an enhancement in generalization rather than in training accuracy. It is worth noting that before switching optimizers, training with the multiple integral term leads to slower convergence or even non-convergence, as the optimizer searches for the flat minima on the loss landscape. However, when considering the entire training process, the epochs taken for convergence to a steady state by Adam and MIAdam are similar.

Appendix D Generalization Proof

Proof of Theorem 1

Proof.

Without loss of generality, we set the hyperparameter $\kappa=1$ and consider the adaptive learning rate at the end of the derivation to make it convenient to analyze the effect of multiple integral term on generalization. Consequently, the parameter update formula for MIAdam1 without adaptive learning rate and switching optimizers is simplified as follows:

\left\{\begin{aligned} &\bm{m}_{t}=\beta_{1}\bm{m}_{t-1}+(1-\beta_{1})\bm{g}_{t},\\ &\overline{\bm{m}}_{t}^{(1)}=\sum^{t}_{j=0}\bm{m}_{j},\\ &\bm{\theta}_{t+1}=\bm{\theta}_{t}-{\alpha}^{2}\overline{\bm{m}}_{t}^{(1)}.\end{aligned}\right.

(18)

Then, we write the deformed motion equation as follows:

\left\{\begin{aligned} &\bm{\mathcal{v}}_{t}=(1-\delta\Delta t)\bm{\mathcal{v}}_{t-1}+\frac{\bm{\mathcal{f}}}{\chi}\Delta t,\\ &\overline{\bm{\mathcal{v}}}_{t}=\sum^{t}_{j=0}\bm{\mathcal{v}}_{j},\\ &\bm{\mathcal{p}}_{t+1}=\bm{\mathcal{p}}_{t}+\overline{\bm{\mathcal{v}}}_{t}{(\Delta t)}^{2},\\ \end{aligned}\right.

(19)

where $\bm{\mathcal{v}}_{t}=-\bm{m}_{t}$ , $\bm{\mathcal{f}}=\bm{g}_{t}$ , $\Delta t$ is equivalent to $\alpha$ , $\delta=(1-\beta_{1})/\Delta t$ , and $\chi=\Delta t/(1-\beta_{1})$ . When the learning rate $\alpha$ is sufficiently small, the differential form of the motion equation is derived as

\displaystyle\chi\frac{\text{d}^{3}\bm{\mathcal{p}}(t)}{\text{d}\tilde{t}^{3}}=\delta\chi\frac{\text{d}^{2}\bm{\mathcal{p}}(t)}{\text{d}\tilde{t}^{2}}+\bm{\mathcal{f}},

(20)

where $\text{d}\tilde{t}=\Delta t$ . As $\bm{\mathcal{f}}$ corresponds to the stochastic gradient term, we obtain

\displaystyle\overline{\chi}\text{d}\dot{\bm{\mathcal{p}}}=\delta\overline{\chi}\text{d}\bm{\mathcal{p}}+\frac{\partial L(\bm{\mathcal{p}})}{\partial\bm{\mathcal{p}}}\text{d}\tilde{t}+2[\psi(\bm{\mathcal{p}})]^{\frac{1}{2}}\text{d}W_{\tilde{t}},

(21)

where $\overline{\chi}=\chi/\tilde{t}$ ; $\text{d}W_{\tilde{t}}\sim\mathscr{N}(0,I\text{d}\tilde{t})$ ; $I$ presents the identity matrix; $\psi(\bm{\mathcal{p}})=\alpha C(\bm{\mathcal{p}})/2$ is the diffusion matrix in the dynamics; $C(\bm{\mathcal{p}})$ denotes the gradient noise covariance matrix. Furthermore, its Fokker-Planck equation in the phase space (the $\dot{\bm{\mathcal{p}}}$ - $\bm{\mathcal{p}}$ space) is written as follows:

$\displaystyle\frac{\partial P(\bm{\mathcal{p}},\bm{\mathcal{v}},\tilde{t})}{\partial\tilde{t}}$	$\displaystyle=-\nabla_{\bm{\mathcal{p}}}\cdot[\bm{r}P(\bm{\mathcal{p}},\bm{\mathcal{v}},\tilde{t})]$	(22)
	$\displaystyle+\nabla_{\bm{r}}\cdot[\delta\bm{\mathcal{v}}+\overline{\chi}\nabla_{\bm{\mathcal{p}}}L(\bm{\mathcal{p}})]$
	$\displaystyle+\nabla_{\bm{\mathcal{v}}}\overline{\chi}^{-2}\psi\cdot\nabla_{\bm{\mathcal{v}}}P(\bm{\mathcal{p}},\bm{\mathcal{v}},\tilde{t})$
	$\displaystyle=-\nabla\cdot\bm{\varphi}(\bm{\mathcal{p}},\tilde{t}),$

where $\bm{r}=\dot{\bm{\mathcal{p}}}$ ; $\bm{\varphi}$ represents the probability flux density; $\nabla\cdot$ denotes the divergence operator. Under Assumption 2, the probability distribution in the valley around $\bm{a}$ is derived as

	$\displaystyle P(\bm{\mathcal{p}},\bm{\mathcal{v}},\tilde{t})$	(23)
$\displaystyle=$	$\displaystyle\int_{\bm{\mathcal{p}}\in V_{\bm{a}}}\text{exp}[-\frac{\delta\overline{\chi}}{2}(\bm{\mathcal{p}}-{\bm{a}})^{\top}(\psi_{\bm{a}}^{-\frac{1}{2}}H_{\bm{a}}\psi_{\bm{a}}^{-\frac{1}{2}})(\bm{\mathcal{p}}-{\bm{a}})]\text{d}V$
$\displaystyle=$	$\displaystyle P({\bm{a}})\frac{(2\pi\delta\overline{\chi})^{2}}{\operatorname{det}(\psi_{\bm{a}}^{-1}H_{\bm{a}})^{\frac{1}{2}}}.$

According to (Kalinay and Percus 2012), in the case of finite inertia, we transform the phase-space equation into the position-space Smoluchowski-like form with the effective diffusion correction:

\displaystyle\hat{\psi}_{i}(\bm{\mathcal{p}}_{i})=\psi_{i}(\bm{\mathcal{p}}_{i})\left(1-\sqrt{1-\frac{4H_{i}(\bm{\mathcal{p}}_{i})}{\delta^{2}\overline{\chi}}}\right)\left(\frac{2H_{i}(\bm{\mathcal{p}}_{i})}{\delta^{2}\overline{\chi}}\right).

(24)

Suppose that $\bm{\varphi}$ is fixed on an escape path from sharp minimum $\bm{a}$ to flat minimum $\bm{b}$ through saddle point $\bm{u}$ , we derive $\bm{\varphi}$ from (Xie et al. 2022) as

	$\displaystyle\bm{\varphi}=$	$\displaystyle\frac{\text{exp}\left(\frac{L({\bm{a}})-L(\bm{u})}{T_{\bm{a}}}P({\bm{a}})\right)}{\int^{\bm{b}}_{\bm{a}}\hat{\psi}^{-1}\text{exp}(\frac{L(\bm{\mathcal{p}})-L({{\bm{\varrho}}})}{T})\text{d}\bm{\mathcal{p}}}$		(25)
	$\displaystyle=$	$\displaystyle\frac{\text{exp}\left(\frac{L({\bm{a}})-L(\bm{u})}{T_{\bm{a}}}P({\bm{a}})\right)}{\hat{\psi}_{\bm{u}}^{-1}\text{exp}\left(\frac{L(\bm{u})-L(\bm{\varrho})}{T_{\bm{u}}}\sqrt{\frac{2\pi T_{\bm{u}}}{\|H_{\bm{u}}\|}}\right)},$		(25)

where $T=\psi/(\delta\overline{\chi})$ and denotes the temperature parameter in the stationary distribution. Based on the formula of probability current and flux, we obtain the flux escaping through saddle point ${\bm{u}}$ :

	$\displaystyle\int_{\bm{s}_{\bm{u}}}\bm{\varphi}\cdot\text{d}\bm{s}$	(26)
$\displaystyle=$	$\displaystyle\varphi_{i}\int_{\bm{s}_{\bm{u}}}\text{exp}\left[-\frac{\delta\overline{\chi}}{2}(\bm{\mathcal{p}}-\bm{u})^{\top}[\psi_{\bm{u}}^{-\frac{1}{2}}H_{\bm{u}}\psi_{\bm{u}}^{-\frac{1}{2}}]^{\bot\bm{e}}(\bm{\mathcal{p}}-\bm{u})\right]\text{d}\bm{s}$
$\displaystyle=$	$\displaystyle\frac{\text{exp}\left(\frac{L(\bm{a})-L(\bm{\varrho})}{T_{{\bm{a}}{\bm{e}}}}\right)P({\bm{a}})\frac{(2\pi\delta\chi)^{\frac{n-1}{2}}}{(\prod_{i\neq e}(\psi_{\bm{u}_{i}}^{-1}H_{\bm{u}_{i}}))^{\frac{1}{2}}}}{\hat{\psi}_{\bm{ue}}^{-1}\text{exp}\left(\frac{L(\bm{u})-L(\bm{\varrho})}{T_{\bm{ue}}}\sqrt{\frac{2\pi T_{\bm{ue}}}{\|H_{\bm{ue}}\|}}\right)},$

where superscript ^⊥e indicates the directions perpendicular to the escape direction $\bm{e}$ . Considering the adaptive learning rate, we have $\psi_{\text{MIAdam1}}=\alpha(\sqrt{[H]^{+})/\mathcal{b}})/2$ according to the case of Adam in (Xie et al. 2022), where superscript ⁺ presents the transformation that $H=U\operatorname{diag}(H_{1},H_{2},\cdots,H_{n-1},H_{n})U^{\top}$ and the $i$ -th column vector of $U$ is the eigenvector corresponding to $H_{i}$ . Finally, we obtain the mean escape time $\phi$ of MIAdam1:

	$\displaystyle\phi_{\text{MIAdam1}}=$	$\displaystyle\frac{P\left(\bm{\mathcal{p}}\in V_{\bm{a}}\right)}{\int_{S_{b}}\bm{\varphi}\cdot\text{d}\bm{s}}$		(27)
	$\displaystyle=$	$\displaystyle\pi\left[\sqrt{1+\frac{4\alpha\sqrt{\mathcal{b}\left\|H_{\bm{u}\bm{e}}\right\|}}{\tilde{t}(1-\beta_{1})}}+1\right]\frac{\left\|\operatorname{det}\left(H_{\bm{a}}^{-1}H_{\bm{u}}\right)\right\|^{\frac{1}{4}}}{\left\|H_{\bm{u}\bm{e}}\right\|}\exp\left[\frac{2\sqrt{\mathcal{b}}\Delta L}{\tilde{t}\alpha}\left(\frac{\varrho}{\sqrt{H_{{\bm{a}}\bm{e}}}}+\frac{(1-\varrho)}{\sqrt{\left\|H_{\bm{u}\bm{e}}\right\|}}\right)\right].$		(27)

The proof is thus completed. ∎

Appendix E Convergence Proof

Definition 1.

For a convex set $F$ , a function $f:\mathbb{R}^{d}\rightarrow\mathbb{R}$ is convex, if for all $\bm{x},\bm{y}\in\mathbb{R}^{d}$ satisfy

f\left(\bm{x}\right)\leq f\left(\bm{x}\right)+\nabla f\left(\bm{x}\right)\left(\bm{y}-\bm{x}\right),

(28)

and for all $\lambda\in[0,1]$ satisfy

f\left(\lambda\bm{x}+\left(1-\lambda\right)\bm{y}\right)\leq\lambda f\left(\bm{x}\right)+\left(1-\lambda\right)f\left(\bm{y}\right).

(29)

Definition 2.

For a convex function $f_{t}$ , we give the $R(\hat{t})$ to determine the convergence. Its expression is shown as follows:

\displaystyle R\left(\hat{t}\right)=\sum_{t=1}^{\hat{t}}f_{t}\left(\bm{\theta}_{t}\right)-\min_{\bm{\theta}}\sum_{t=1}^{\hat{t}}f_{t}\left(\bm{\theta}\right).

(30)

When $\lim_{\hat{t}\rightarrow\infty}R\left({\hat{t}}\right)/{\hat{t}}\neq 0$ , we consider the algorithm is not convergent.

Proof of Theorem 2

Proof.

According to the Definition 1 and the Definition 2, we have

	$\displaystyle R\left({\hat{t}}\right)=$	$\displaystyle\sum_{t=1}^{\hat{t}}[f_{t}\left(\bm{\theta}_{t}\right)-f_{t}\left(\bm{\theta}^{*}\right)]$		(31)
	$\displaystyle\leq$	$\displaystyle\sum_{i=1}^{d}\sum_{t=1}^{\hat{t}}g_{t,i}\left(\theta_{t,i}-\theta_{,i}^{*}\right).$		(31)

From the updating rules of MIAdam, we get

$\displaystyle\theta_{t+1,i}=$	$\displaystyle\theta_{t,i}-\alpha\frac{\hat{m}^{(1)}_{t,i}}{\sqrt{\hat{v}_{t,i}}}$	(32)
$\displaystyle=$	$\displaystyle\theta_{t,i}-\alpha\frac{1}{1-\beta_{1}^{t}}\frac{\overline{m}^{(1)}_{t,i}}{\sqrt{\hat{v}_{t,i}}}$
$\displaystyle=$	$\displaystyle\theta_{t,i}-\alpha\frac{1}{1-\beta_{1}^{t}}\frac{\sum^{t}_{r=1}\kappa_{1}^{t-r}m_{r}}{\sqrt{\hat{v}_{t,i}}}.$

Then we obtain

	$\displaystyle g_{t,i}\left(\theta_{t,i}-\theta_{,i}^{*}\right)$	(33)
$\displaystyle=$	$\displaystyle\underbrace{\frac{(1-\beta_{1}^{t})\sqrt{\hat{v}_{t,i}}}{2\alpha\left(1-\beta_{1,t}\right)}\left(\left(\theta_{t,i}-\theta_{,i}^{}\right)^{2}-\left(\theta_{t+1,i}-\theta_{,i}^{}\right)^{2}\right)}_{\{1\}}$
$\displaystyle+$	$\displaystyle\underbrace{\frac{\beta_{1,t}m_{t-1,i}\left(\theta_{,i}^{}-\theta_{t,i}\right)}{\left(1-\beta_{1,t}\right)}}_{\{2\}}+\underbrace{\frac{\kappa_{1}\overline{m}^{(1)}_{t-1,i}\left(\theta_{,i}^{}-\theta_{t,i}\right)}{\left(1-\beta_{1,t}\right)}}_{\{3\}}$
$\displaystyle+$	$\displaystyle\underbrace{\frac{\alpha(1-\beta_{1}^{t})}{2\left(1-\beta_{1,t}\right)}\frac{\hat{m}_{t,i}^{2}}{\sqrt{\hat{v}_{t,i}}}}_{\{4\}}.$

For the first two terms {1} and {2} in Eq. (LABEL:eq:23), we derive the following two inequalities based on (Kingma and Ba 2015):

	$\displaystyle\sum_{i=1}^{d}$	$\displaystyle\sum_{t=1}^{\hat{t}}\frac{(1-\beta_{1}^{t})\sqrt{\hat{v}_{t,i}}}{2\alpha\left(1-\beta_{1,t}\right)}\left(\left(\theta_{t,i}-\theta_{,i}^{}\right)^{2}-\left(\theta_{t+1,i}-\theta_{,i}^{}\right)^{2}\right)$		(34)
	$\displaystyle\leq$	$\displaystyle\frac{\mathsf{d}^{2}}{2\alpha(1-\beta_{1})}\sum^{d}_{i=1}\sqrt{{\hat{t}}\hat{v}_{{\hat{t}},i}},$		(34)

and

	$\displaystyle\sum_{i=1}^{d}$	$\displaystyle\sum_{t=1}^{\hat{t}}\frac{\beta_{1,t}m_{t-1,i}\left(\theta_{,i}^{*}-\theta_{t,i}\right)}{\left(1-\beta_{1,t}\right)}$			(35)
	$\displaystyle\leq$	$\displaystyle\frac{\alpha(1+\beta_{1})\mathsf{g}_{\infty}}{(1-\beta_{1})\sqrt{1-\beta_{2}}(1-\gamma)^{2}}\sum^{d}_{i=1}\lVert g_{1:{\hat{t}},i}\rVert_{2}+$	$\displaystyle\sum_{i=1}^{d}\frac{\mathsf{d}^{2}_{\infty}\mathsf{g}_{\infty}\sqrt{1-\beta_{2}}}{2\alpha\beta_{1}(1-\lambda)^{2}}.$		(35)

Since the gradient is assumed to be bounded, the $m_{t,i}$ should also be bounded in Adam and satisfies $m_{t,i}\leq\mathsf{g}$ . For the term {3}, we have

	$\displaystyle\sum^{\hat{t}}_{t=2}\frac{\kappa_{1}\overline{m}^{(1)}_{t-1,i}\left(\theta_{,i}^{*}-\theta_{t,i}\right)}{\left(1-\beta_{1,t}\right)}$	(36)
$\displaystyle=$	$\displaystyle\sum^{\hat{t}}_{t=2}\frac{\left(\theta_{,i}^{*}-\theta_{t,i}\right)}{\left(1-\beta_{1,t}\right)}\sum^{t-1}_{r=1}\kappa_{1}^{t-r}m_{r}$
$\displaystyle\leq$	$\displaystyle\sum^{\hat{t}}_{t=2}\frac{\left(\theta_{,i}^{*}-\theta_{t,i}\right)}{\left(1-\beta_{1,t}\right)}\left(\frac{\kappa_{1}}{1-\kappa_{1}}-\frac{\kappa^{t}}{1-\kappa_{1}}\right)t\mathsf{g},$

which is a divergent series. Therefore, we further deduce the following limit:

\displaystyle\lim_{{\hat{t}}\rightarrow\infty}\frac{\sum_{i=1}^{d}\sum^{\hat{t}}_{t=2}\frac{\kappa_{1}\overline{m}^{(1)}_{t-1,i}\left(\theta_{,i}^{*}-\theta_{t,i}\right)}{\left(1-\beta_{1,t}\right)}}{{\hat{t}}}\rightarrow\infty.

(37)

For the term {4}, we have

		$\displaystyle\sum_{i=1}^{d}\sum^{\hat{t}}_{t=1}\frac{\alpha(1-\beta_{1}^{t})}{2\left(1-\beta_{1,t}\right)}\frac{\hat{m}_{t,i}^{2}}{\sqrt{\hat{v}_{t,i}}}$		(38)
	$\displaystyle=$	$\displaystyle\sum_{i=1}^{d}\sum^{\hat{t}}_{t=1}\frac{\alpha(1-\beta_{1}^{t})}{2\left(1-\beta_{1,t}\right)}\frac{(\sum^{t-1}_{r=1}\kappa_{1}^{t-r}m_{r})^{2}}{\sqrt{\hat{v}_{t,i}}}.$		(38)

Analogous to the derivation of the term {3}, we similarly obtain the following limit:

\displaystyle\lim_{{\hat{t}}\rightarrow\infty}\frac{\sum_{i=1}^{d}\sum^{\hat{t}}_{t=1}\frac{\alpha(1-\beta_{1}^{t})}{2\left(1-\beta_{1,t}\right)}\frac{\hat{m}_{t,i}^{2}}{\sqrt{\hat{v}_{t,i}}}}{{\hat{t}}}\rightarrow\infty.

(39)

Finally, following the above derivation, we have

\displaystyle\lim_{{\hat{t}}\rightarrow\infty}\frac{R({\hat{t}})}{{\hat{t}}}\neq 0.

(40)

The proof is thus completed. ∎