Enhancing sharp augmented Lagrangian methods with smoothing techniques for nonlinear programming

We want to solve the following nonlinear programming problem:

where $f:\mathbbm{R}^{n}\to\mathbbm{R}$ and $h:\mathbbm{R}^{n}\to\mathbbm{R}^{m}$ are twice continuously differentiable. Stationary points $x$ of (1) along with the associated Lagrange multipliers $\lambda$, are characterized by the following system of equations

where $L:\mathbbm{R}^{n}\times\mathbbm{R}^{m}\to\mathbbm{R}$ such that $L\left(x,\lambda\right)=f\left(x\right)+\left\langle\lambda,h\left(x\right)\right\rangle$, is the Lagrangian function associated with problem (1).

Augmented Lagrangian methods have been extensively used to solve nonlinear programming problems. These methods involve the iterative minimization of an augmented Lagrangian function with respect to the primal variables, followed by appropriate updates to the dual variables. Among the various methods, the most studied are the so–called Powell–Hestenes–Rockafellar (PHR) augmented Lagrangian methods hestenes1969multiplier ; powell1969method ; rockafellar1974augmented , which are based on the function $\bar{L}_{2}:\mathbbm{R}^{n}\times\mathbbm{R}^{m}\times\left(0,\infty\right)\to\mathbbm{R}$ such that

This method has been studied extensively in the literature bertsekas1982constrained ; conn1991globally ; conn1996convergence ; pennanen2002local ; birgin2005numerical ; andreani2008augmented ; fernandez2012local ; birgin2014practical , and implemented successfully in software packages such as LANCELOT conn1992lancelot and ALGENCAN andreani2007augmented . We emphasize that at each iteration, the function to be minimized, $x\mapsto\bar{L}_{2}\left(x;\lambda,r\right)$, is continuously differentiable. Furthermore, convergence results can be obtained even if, instead of a global minimizer, an approximate stationary point is found at each iteration.

Another group of methods is based on the sharp augmented Lagrangian function $\bar{L}_{1}:\mathbbm{R}\times\mathbbm{R}^{m}\times\left(0,\infty\right)\to\mathbbm{R}$ defined as follows:

Methods based on this function have been studied in gasimov2002augmented ; burachik2006modified ; jefferson2009thesis ; burachik2010primal ; kasimbeyli2009modified ; bagirov2019sharp . These studies propose a duality scheme that preserves the main idea of the modified subgradient algorithm, where at each iteration, a function of the dual variables is obtained by globally minimizing $x\mapsto\bar{L}_{1}\left(x;\lambda,r\right)$, and the dual variables are updated in the direction of a subgradient of the dual function. A practical drawback of these methods is that at each iteration, one must find a global minimizer, or a good approximation, of a nonlinear, nonconvex, and nondifferentiable function.

In this work, we propose a method based on the sharp augmented Lagrangian with a primal approach similar to that used in PHR augmented Lagrangian methods. To perform the primal minimization, we shall use a suitable smoothing technique. Among all possible approaches to smoothing out the kinks of the sharp augmented Lagrangian function, we choose the one introduced in fernandez2022augmented . Specifically, it was shown that

This relation establishes the sharp augmented Lagrangian as a scalarization of the penalty parameter in the PHR augmented Lagrangian (3). By adopting this approach, we aim to inherit some of the desirable properties of the PHR augmented Lagrangian.

For parameters $\left(\lambda,r\right)\in\mathbbm{R}^{m}\times\left(0,\infty\right)$, we define the extended–real–valued smoothing function as follows:

It can be observed that $\tilde{L}_{\lambda,r}$ is a lower semicontinuous function on $\mathbbm{R}^{n}\times\mathbbm{R}$ and is continuously differentiable on $\mathbbm{R}^{n}\times\left(0,\infty\right)$. Clearly, for this function we have $\bar{L}_{1}\left(x;\lambda,r\right)=\min_{t\geq 0}\left\{\tilde{L}_{\lambda,r}\left(x,t\right)\right\}$.

The goal is to modify the sharp augmented Lagrangian method by replacing the minimization of the function $x\mapsto\bar{L}_{1}\left(x;\lambda,r\right)$ with the minimization of the function $\left(x,t\right)\mapsto\tilde{L}_{\lambda,r}\left(x,t\right)$. The nondifferentiability of $\bar{L}_{1}$ at $x$ where $h\left(x\right)=0$ is addressed by introducing a singularity in $\tilde{L}_{\lambda,r}$ at $t=0$. We will demonstrate that employing this smoothing technique does not compromise the classical duality scheme, which requires exact minimizers at each step. Additionally, we will present a primal approach where stationary points at each step are acceptable.

The rest of the paper is organized as follows. In Section 2, we examine the classical duality approach based on exact minimization for the smoothing function. Section 3 introduces the primal approach using stationary points, detailing two algorithms: one employing standard descent and the other using coordinate descent. Section 4 studies the boundedness of the penaly parameter. Section 5 provides a comparison between the PHR augmented Lagrangian method and our two primal algorithms. Section 6 is dedicated to conclusiones, and finally, an Appendix is included, describing a set of test problems chosen by the authors.

We conclude this section by defining our notation. We use $\langle\cdot,\cdot\rangle$ to denote the Euclidean inner product and $\|\cdot\|$ to represent the associated norm.

We will prove that finding the global minimizer of $\tilde{L}_{\lambda,r}$ instead of the global minimizer of $\bar{L}_{1}\left(\cdot;\lambda,r\right)$ allows us to recover the main results from (jefferson2009thesis, , Chapter 2). To achieve this, we propose a method using a dual approach based on a modified subgradient algorithm. To this end, we refer to (1) as the primal problem and define its associated augmented dual problem:

where

Let us denote the set of minimizers of $\tilde{L}_{\lambda,r}$ by $A$, that is,

For the exact case, we will use Algorithm 1, which follows the classical dual approach based on the modified subgradient algorithm.

In the next result we show some properties of the elements within the solution set $A$. One of these properties reveals a relationship between the minimizers of $\tilde{L}_{\lambda,r}$ and the subgradients of $-\tilde{q}$. To establish this, note first that $-\tilde{q}$ is a convex function, as it is the supremum of affine functions. Therefore, its subdifferential is given by

Now we will study the sequences generated by Algorithm 1. First, note that if the smoothing parameter is zero, the primal point is feasible, since by Lemma 1(a), we have $\left\|h\left(x^{k}\right)\right\|=t_{k}=0$. Also, the dual updates follow a subgradient direction of $-\tilde{q}$, i.e., a proximal point iteration of a linearization of $-\tilde{q}$. Note that the problem

has the unique solution

Clearly, Algorithm 1 will generate a sequence if at Step 1 a global minimizer of $\tilde{L}_{\lambda^{k},r_{k}}$ is found. The existence of such a minimizer can be ensured under various sets of assumptions. Since our result is independent of such assumptions, we will simply assume that $A\left(\lambda^{k},r_{k}\right)\neq\emptyset$.

The next result states that if the algorithm stops, we obtain both a primal and a dual solution, otherwise the generated dual variables will approach the maximum of $\tilde{q}$.

Boundedness of the sequences generated by an algorithm is crucial for analyzing convergence. The following lemma describes what happens when the sequence of multipliers or the sequence of penalty parameters is bounded.

The next lemma provides a sufficient condition for the boundedness of the sequence of penalty parameters.

The following theorem guarantees the finite termination of Algorithm 1, provided that dual solutions exist.

It is important to emphasize that in the case of no duality gap, i.e., when $\tilde{q}\left(\lambda,r\right)=f\left(x\right)$ for feasible dual and primal points $\left(\lambda,r\right)$ and $x$, respectively, the vector $\lambda$ is considered as a Lagrange multiplier in an extended sense. The structure of our augmented dual function enables us to show the existence of a Lagrange multiplier that satisfies (2).

Note that Step 1a of Algorithm 1 requires to tackle two issues:

• •

  dealing with a nondifferentiable objective function when $t=0$,

• •

  finding an exact global minimizer.

Regarding the nondifferentiability, it is possible that all solutions are of the form $\left(x,0\right)$ where $\tilde{L}_{\lambda,r}$ is not continuous and hence not differentiable. Additionally, since most optimization solvers assume some degree of smoothness for the objective function and constraints, numerical issues may arise.

Furthermore, global optimization algorithms tend to be slow or unreliable for large– or medium– scale problems birgin2014practical . Moreover, exact solutions are unattainable due to the limitations of finite arithmetic in computers.

In preliminary implementations of Algorithm 1, we observed that Step 1a could not finish satisfactorily when $t_{k}$ was close to zero and the penalty parameter $r_{k}$ became excessively large.

In the next section we propose a strategy to address the nondifferentiability at $t=0$. We also relax the requirement of finding an exact solution to the subproblem, allowing for an inexact stationary point.

In this section, we present a primal approach to the previous algorithm. The main advantage of this approach is that it allows us to replace exact minimizers with inexact stationary points. It is well known that for nonsmooth functions, such as $x\mapsto\left|x\right|$, there may be no points in a neighborhood of the minimizer where the derivative of the function is close to zero. This behavior can also be observed in $\tilde{L}_{\lambda,r}$ when applied to simple problems, as illustrated in the next example.

To ensure the existence of inexact stationary points for a continuously differentiable function, we introduce a barrier function associated with the condition $t\geq 0$. For computational simplicity, we employ the inverse barrier function. The function to be minimized is defined as:

where $s>0$ is the barrier parameter. This function is clearly lower semicontinuous and continuously differentiable for $t>0$. The existence of inexact stationary points is shown in the following proposition.

Recall that $\nabla\hat{L}^{s}_{\lambda,r}\left(x,t\right)=\left(\nabla_{x}\hat{L}^{s}_{\lambda,r}\left(x,t\right),\frac{\partial\hat{L}^{s}_{\lambda,r}}{\partial t}\left(x,t\right)\right)$ where

These expressions will be useful in the forthcoming calculations.