Strong contraction mapping and topological non-convex optimization

Calculus of variation plays a critical role in the modern calculation. Mostly, the function of interest cannot be obtained directly but makes a functional attain its extremum. This leads people to construct different functionals according to different questions, such as the least action principle, Fermat’s principle, maximum likelihood estimation, finite element analysis, machine learning and so forth. These methods provide us the routine to transfer the original problem to an optimization problem, which makes optimization methods have been used almost everywhere.

However, the popular gradient-based optimization methods be applied to a non-convex function with many local minima is unprincipled. They face great challenges of finding the global minimum point of the function. Because the information from the derivative of a single point is not sufficient to decide the global geometrical property of the function. In the gradient-based methods, the domain of the searching area is divided into several subsets with regards to different local minima. And usually, iterating points will converge to one local minimum depends on where the initial point starts from. For a successful minimum point convergence, the initial point luckily happens to be sufficiently near the global minimum point.

It is the very time to think outside box and try to cultivate some method other than gradient-based methods. Before that, let us first think about why gradient-based optimization methods fail the task of global minimum convergence. Let (X,d) be a metric space and T:X $\rightarrow$ X is a self-mapping. For the inequality that,

if $q\in[0,1),$ T is called contractive; if $q\in[0,1],$ T is called nonexpansive; if $q<\infty$, T is called Lipschitz continuousHusain ; Latif . The gradient-based methods are usually nonexpansive mapping. The fixed point may exist but is not unique for a general situation. For instance, if the objective function $f(x)$ has many local minima, for the gradient descent method mapping $T(x)=x-\gamma\nabla f(x)$, any local minimum is a fixed point accordingly. From the perspective of spectra of a bounded operator, for a nonexpansive mapping, any minimum of the objective function is an eigenvector of the eigenvalue equation $T(x)=\lambda x$ , in which $\lambda=1$. In the optimization problem, nonexpansive mapping sometimes works but their weakness is obvious.

It is worth to note that to summarize the condition for the existence of fixed-point of non-expansive mapping or Lipschitz continuous mapping is a great challenge. Topologically, to prove the existence of fixed-point of the contraction mapping is relatively easier than to prove the existence of fixed-point of non-expansive mapping or Lipschitz continuous mapping. For contraction mapping, the set of the range is shrinking. While, for non-expansive mapping or Lipschitz continuous mapping, the change of range set can involve complicated transformation and rotationAhues ; Rudin . There is no straightforward way to find the fixed-point of mapping. The fixed-point(or fixed-points) of Lipschitz continuous mapping, even if it exists, can cunningly hide inside an inflating volume. Because both the existence and uniqueness of solution are important in optimization problem so that the contractive mapping is more favored than the nonexpansive mapping.

The well-known Banach fixed-point theorem, as the first fixed-point theorem on contraction mapping, plays an important role in solving linear or nonlinear system. But for optimization problems, the condition of contraction mapping $T:X\rightarrow X$ that $d(T(x),T(y))\leq qd(x,y)$, which usually requires convexity of the function, is hard to be applied to non-convex optimization problem. In this study, we are trying to extend the Banach fixed-point theorem to an applicable method for optimization problems, which is called strong contraction mapping.

Strong contraction mapping is a surjection self-mapping that always maps to the subset of its domain. We will prove that strong contraction mapping admits a unique fixed-point, explain how to build an optimization method as an application of strong contraction mapping and illustrate why its fixed-point is the global minimum point of the objective function.

Recall the definition of diameter $D(X)$ of a metric space $X$.

This contraction mapping is called strong because the requirement $D(X_{i+1})\leq q(X_{i})$ is looser than $d(T(x),T(y))\leq qd(x,y)$ what in Banach fixed-point theorem, which doesn’t require the distance between two points getting smaller and smaller but the diameter of the range of the mapping getting smaller and eventually shrinking to a point. Thereafter, the inequality $d(T(x),T(y))\leq qd(x,y)$ is included by the inequality $D(X_{i+1})\leq q(X_{i})$ as a special case.

To prove the Theorem.1, one can follow the same logic of Banach fixed-point theorem proofBanach but substitute the inequality $d(T(x),T(y))\leq qd(x,y)$ with $D(X_{i+1})\leq q(X_{i})$. Let $x_{0}\in X$ be arbitrary initial point and define a sequence $\{x_{i}\}$ as: $x_{i}=T(x_{i-1})$.

Proof.Let $m,n\in\mathbb{N}$ such that $m>n$.

Let $\epsilon>0$ be arbitrary, since $q\in[0,1)$, we can find a large $N\in\mathbb{N}$ such that

Hence, by choosing $m,n$ large enough:

Thus, $\{x_{i}\}$ is Cauchy sequence. $\Box$

As long as $D(X_{0})$ is bounded, the convergence is guaranteed, which is independent of the choice of $x_{0}$.

Proof.

Thus,$x^{*}=T(x^{*})$. $\Box$

Proof. Suppose there exists another fixed-point y that $T(y)=y$, then choose the subspace $X_{i}$ that both the $x^{*}$ and $y$ are the only elements in $X_{i}$. By definition, $X_{i+1}=\mathcal{R}(T(X_{i}))$ so that, both the $x^{*}$ and $y$ are elements in $X_{i+1}$, namely,

Since $(X,d)$ is a non-empty complete metric space and the diameter $D(X^{*})=0$, $X^{*}$ has a single element $x^{*}$.

Thus $x^{*}=y$. $\Box$

In this section, we have proven the existence and uniqueness of fixed-point of strong contraction mapping. Compared with contraction map, the strong contraction map can address a much wider range of problems as the requirement $D(T(x_{i}))\leq D(x_{i})$ is looser than $d(T(x),T(y))\leq d(x,y)$. Different from $d(T(x),T(y))\leq d(x,y)$, the inequality $D(T(x_{i}))\leq D(x_{i})$ doesn’t require $x_{i}$ in sequence $\{x_{n}\}$ must move closer to each other for every step but confine the range of mapping to be smaller and smaller. Thereafter, the sequence $\{x_{n}\}$ is Cauchy sequence and converge to the fixed-point. It is worth to mention if some mapping $T$ is not a strong contraction mapping. Still, one can think about whether $T^{n}$ or $T^{-n}$ is a strong contraction mapping to find a way around. If that is the case, then the iteration of $T^{n}$ or $T^{-n}$ yields a fixed-point.

After the discussion of strong contraction mapping, let us think about how to construct an optimization algorithm occupied with the property of strong contraction mapping.

The objective function $f$ is a mapping defined on X, $f:X\rightarrow R$.

To overcome the dilemma that the optimization method may be saturated by local minima, intuitively, one can utilize a hyperplane that parallels to the domain to cut the objective function $f$ such that the intersection between the hyperplane and the function are contours. They reflect the global geometrical property of the function. The difficulty is how to iterate the hyperplane to move downwards and decide the position of the global minimum point.

Observing that contours will divide the objective function $f$ into two parts. One is higher than the height of contours and the other is lower than the height of contours. Now, our task is to map to a point lower than the height of contours. As a numerical method, instead of getting all points of contours or symbolic expression of contours, we want to get n roots on the contours using root finding algorithm, that is

where, $i$ indicates $i$th iterate and $j$ indicate $j$th root.

First, provide one arbitrary initial point $x_{0}$ to the function and calculate the height of contours $f(x_{0})=L_{0}$ at the height of the initial point;

If the objective function $f$ is non-convex, as a consequence, to map to a point lower than the height of contours cannot be achieved by simply averaging the roots on contours. However, the objective function $f$ mostly can be decomposed into many locally convex subsets. We can map to a point lower than the height of contours by averaging the roots belong to the same locally convex subsets and iterate hyperplane downwards.

A function $g:\mathcal{R}^{n}\rightarrow\mathcal{R}$ is convex, if for every $x,y\in\mathcal{R}^{n}$ and $\lambda\in[0,1]$, we have the inequality Bertsekas

The inequality $d(x_{m},x_{n})\leq q^{n}D(X_{0})$ indicates the rate of convergence. The smaller q is, the higher rate of convergence is achieved. To achieve high rate of convergence, it is important to extend the Inequality.4 to the situation with regards to many points $r_{1},r_{2},...,r_{n}$ ,

By induction,

which is Jensen’s Inequality Chandler ,

where,

When we apply the Jensen’s Inequality.6 to the objective function $f$. Let $\lambda_{j}=\frac{1}{n}$,for $j=1,2,...,n$, the Jensen’s Inequality.6 turns to be a strict inequality, that is

Let $x_{i+1}=\frac{1}{n}\sum_{j=1}^{n}r_{j}^{i}$ , that is

Therefore, it is important to check whether two roots belong to the same locally convex subset and traverse all roots, decompose them into these locally convex subsets and get averages accordingly. Based on the Inequality.4, one practical way to achieve that is to pair every two roots on contours and scan function’s value along the segment between the two roots and check whether there exists a point higher than contours’ height. Traverse all the roots and apply this examination on them, then we can decompose the roots with regards to different locally convex subsets. To check whether two roots belong to the same convex subset, $N$ number of random points along the segment between two roots are checked whether higher than the contour’s level or notSchachter ; Tseng . If the Inequality.4 always holds for $N$ times of test, then we believe the two roots locate at the same locally convex subset and store them together.

After the set of roots being decomposed into several locally convex subsets, the average of roots in the same locally convex subset is always lower than the contours’ height due to Jensen’s Inequality.6.

Since the iterating point $x_{i+1}$ is always lower than $x_{i}$ ,

Hence, there exists a $\xi\in(0,1)$ such that,

As $i$ goes to infinity, then

Because the fixed-point $x^{*}$ is at the same height as the global minimum point $x_{min}$ and the global minimum point is unique. Thus, the fixed-point $x^{*}$ coincides with the global minimum point. The iteration $x_{i+1}=T{x_{i}}$ yields the fixed-point $x^{*}$that solves the equation $T(x^{*})=x^{*}$.Taylor

Rewrite the mapping that $x_{i+1}=T(x_{i})$,which is averaging of roots $r_{1},r_{2},...r_{n}$ that locate at the same locally convex subset,

Then, like a Russian doll, the dynamical system can be explicitly written as an expansion of iterates as

After the set of roots are decomposed into several convex subsets, the averages of roots with regards to each subset are calculated and the lowest one is returned as an update point from each iterate. Thereafter, the remaining calculation is to repeat the iterate over and over until convergence and return the converged point as the global minimum. The decomposition of roots set provided a divide-and-conquer method to transfer the original problem to a number of subproblems and solve them recursively. Cormen

In summary, the main procedure of topological non-convex optimization method is decomposed into the following steps: 1. Given the initial guess point $x_{0}$ for the objective function and calculate the contour level $L_{0}$; 2. Solve the equation $f(x_{i})=L_{i}$ and get $n$ number of roots. Decompose the set of roots into several convex subsets, return the lowest average of roots as an update point from each iterate; 3. Repeat the above iterate until convergence.

According to the previous discussion, the mechanism of the proposed optimization method has posted some restrictions to the objective function. In summary, it requests the objective function to be continuous, occupied with a unique global minimum point, can be decomposed into many locally convex components and its domain is non-empty complete. And the good news is the most of real-world non-convex optimization problems can satisfy these prerequisites.

First of all, the optimization algorithm has been tested on a convex function the Sphere function $f(\boldmath{x})=\sum{x^{2}_{i}}$. The minimum is (0,0,0), where $f(0,0,0)=0$. The iterations of roots and contours is shown in FIG.1 and the update of searching point is shown in TABLE.1.

Then, we test the optimization algorithm on the McCormick function. And the first 4 iterates of roots and contour is shown in FIG.2 and the detailed iteration of the searching point from the numerical calculation is shown in TABLE.2. The test result indicate the average of roots can move towards the global minimum point (-0.54719,-1.54719), where $f(-0.54719,-1.54719)=-1.9133$.

Ackley function would be a nightmare for most gradient-based methods. Nevertheless, the algorithm has been tested on Ackley function where the global minimum locates at (0,0) that $f(0,0)=0$. And the first 6 iterates of roots and contours are shown in FIG.3 and the minimum point (-0.00000034,0.00000003) return by the algorithm is shown in TABLE.3. The test result shows that the optimization algorithm is robust to local minima and able to achieve the global minimum convergence. The quest to locate the global minimum pays off handsomely.

The observation from these experiments is that the size of contour become smaller and smaller during the iterative process and eventually converge to a point, which is the global minimum point of the function.

We introduced the definition of the strong contraction mapping and the existence and uniqueness of its fixed-point in this paper. As an extension of Banach fixed-point theorem, the iteration of strong contraction mapping is a Cauchy sequence and yields the unique fixed-point, which perfectly fit with the task of optimization. The global minimum convergence regardless of local minima and initial point position is a very significant strength for the optimization algorithm. And we illustrated how to implement an optimization method occupied with strong contraction mapping property. This topological optimization method finds a way around that even if the objective function is non-convex, still, we can decompose it into many convex components and take advantage of convexity of a locally convex component to pin down the global minimum point. This optimization method has been tested on Sphere, McCormick and Ackley functions and successfully achieved the global minimum convergence as expected. These experiments demonstrate the contours’ shrinking and the iterating point’s approaching the global minimum point. We look forward to extending our study to the higher dimensional situation and believe that the optimization method works for that in principle.