Bad semidefinite programs: they all look the same

Many problems in engineering, combinatorial optimization, machine learning, and related fields can be formulated as the primal-dual pair of conic linear programs

where $\mathcal{A}:X\rightarrow Y$ is a linear map between finite dimensional Euclidean spaces $X$ and $Y,\,$ $\mathcal{A}^{*}$ is its adjoint, $K\subseteq Y$ is a closed, convex cone, $K^{*}$ is its dual cone, and $s\leq_{K}t\,$ means $t-s\in K.\,$ Note that the subscript $c$ refers to the objective of the primal problem.

Problems $(\mathit{P_{c}})$ and $(\mathit{D_{c}})$ generalize linear programs and share some of the duality theory of linear programming. For instance, a pair of feasible solutions always satisfies the weak duality inequality $\langle c,x\rangle\leq\langle b,y\rangle.$ However, in conic linear programming pathological phenomena occur: the optimal values of $(\mathit{P_{c}})$ and of $(\mathit{D_{c}})$ may differ, and they may not be attained.

In particular, semidefinite programs (SDPs) and second order conic programs (SOCPs) — probably the most useful and pervasive conic linear programs — often behave pathologically: for a variety of examples we refer to the textbooks [6, 37, 11, 3, 42] surveys [44, 41, 27] and research papers [34, 1, 43]. Pathological conic LPs are both theoretically interesting and often difficult, or impossible to solve numerically.

These pathologies arise, since the linear image of a closed convex cone is not always closed. For recent studies about when such sets are closed (or not), see e.g., [4, 2, 29]. Three approaches (which we review in detail below) can help to avoid or remedy the pathologies: one can impose a constraint qualification (CQ), such as Slater’s condition; one can regularize $(\mathit{P_{c}})\mathchar 45\relax(\mathit{D_{c}})$ using a facial reduction algorithm [16, 46, 31]; or write an extended dual [34, 24], which uses extra variables and constraints. However, such CQs often do not hold, and neither facial reduction algorithms nor extended duals can help solve all pathological instances.

We started this research observing that pathological SDPs in the literature look curiously similar and one of our main goals is to find the root cause of the similarity. We focus on the system underlying $(\mathit{P_{c}})$ and call

badly behaved if there exists $c$ such that $(\mathit{D_{c}})$ either does not attain its value or its value differs from the value of $(\mathit{P_{c}}).$ We call ($P$) well behaved if it is not badly behaved.

We illustrate our results by many examples. The only technical proofs in the main body of the paper are those of Theorem 1 and of Lemma 5, and these can be safely skipped at first reading.

We say that $(\mathit{D_{c}})$ is a strong dual of $(\mathit{P_{c}})$ if they have the same value, and $(\mathit{D_{c}})$ attains this value, when it is finite. Thus in general $(\mathit{D_{c}})$ is not a strong dual of $(\mathit{P_{c}}).$ Using this terminology, ($P$) is well behaved exactly if $(\mathit{D_{c}})$ is a strong dual of $(\mathit{P_{c}})$ for all $c.$ We say that ($P$) satisfies Slater’s condition, if there is $x$ such that $b-\mathcal{A}x$ is in the relative interior of $K;$ if this condition holds, then ($P$) is well behaved.

Ramana in [34] proposed a strong dual for SDPs, which uses polynomially many extra variables and constraints. His result implies that semidefinite feasibility is in ${\cal NP}\cap{\rm co\mathchar 45\relax}{\cal NP}$ in the real number model of computing. Klep and Schweighofer in [24] constructed a Ramana-type strong dual for SDPs, which, interestingly, is based on ideas from algebraic geometry, rather than from convex analysis.

The facial reduction algorithm of Borwein and Wolkowicz in [16, 15] converts ($P$) into a system that satisfies Slater’s condition, and is hence well behaved. The algorithm relies on a sequence of reduction steps. For more recent, simplified facial reduction algorithms, see Waki and Muramatsu [46] and Pataki [31]. Ramana, Tunçel, and Wolkowicz in [36] proved the correctness of Ramana’s dual from the facial reduction algorithm of [16, 15], showing the connection of these two seemingly unrelated concepts. We refer to Ramana and Freund [35] for a proof that the Lagrange dual of Ramana’s dual has the same value as the original problem. Generalizations of Ramana’s dual are known for conic LPs over nice cones [31]; and for conic LPs over homogeneous cones (Pólik and Terlaky [33]).

For a generalization of the concept of strict complementarity (a concept that plays an important role in our work), we refer to Pena and Roshchina [32]. Schurr et al in [40] characterize universal duality — when strong duality holds for all right hand sides and objective functions. Tunçel and Wolkowicz in [43] related the lack of strict complementarity in a homogeneous conic linear system to the existence of an objective function with a positive gap.

We finally remark that the technique of reformulating equality constrained SDPs (relying mostly on elementary row operations), to easily verify their infeasibility was used recently by Liu and Pataki [25].

In this section we present our main characterization of when ($P$) is badly or well behaved (these concepts are defined in the Introduction). We first need a definition.

We start with a basic lemma:

∎

To put Lemma 2 into perspective, note that the image set in Lemma 2 is closed if $(\mathcal{A},b)^{*}K^{*}$ is closed (one can argue this directly or by modifying the proof of Lemma 2). In turn, if $(\mathcal{A},b)^{*}K^{*}$ is closed, then the duality gap between $(\mathit{P_{c}})$ and $(\mathit{D_{c}})$ is zero, even if $K$ lives in an infinite dimensional space — see, e.g., Theorem 7.2 in [3] (where our primal is called the dual). The proof of Lemma 2 is standard, and we give it in Appendix B.

The main result of this section follows (recall the definition of $\operatorname{dir}(z,K)$ and related sets from (1.4)–(1.6)). We write ${\cal R}({\cal A},b)$ for the rangespace of the operator $(x,t)\rightarrow{\cal A}x+bt.$

∎

To build intuition we show how Theorem 1 unifies two classical, seemingly unrelated, sufficient conditions for ($P$) to be well-behaved.

Though Lemma 2 is a bit simpler to state than Theorem 1, the latter will be more useful. On the one hand, Lemma 2 relies on the closedness of the linear image of $K^{*}\times\mathbb{R}_{+},\,$ which may not be easy to check. On the other hand, Theorem 1 relies on the geometry of the cone $K\,$ itself, and not on the geometry of its linear image. The geometry of typical cones that occur in optimization — e.g. the geometry of the semidefinite cone — is well understood. Thus Theorem 1, among other things, will lead to a proof that badly behaved semidefinite systems are in ${\cal NP}\cap{\rm co\mathchar 45\relax}{\cal NP}$ in the real number model of computing. Lemma 2, by itself, affords no such corollary.

Note that if $b=0$, then by Lemma 2 the system ($P$) is well behaved iff $\mathcal{A}^{*}K^{*}$ is closed. Thus in this case Lemma 1 and Theorem 1 are equivalent. To prove the general case of Theorem 1 we use a homogenization argument.

Claim There is a $z$ maximum slack in ($P$) such that $(z,1)$ is also a maximum slack in ($P_{h}$).

To prove the Claim we first note that if $z$ is a maximum slack in ($P$) and $z^{\prime}$ is some other slack, then $\lambda z+(1-\lambda)z^{\prime}$ is also a maximum slack for all $0<\lambda\leq 1\,$ (by Theorem 6.1 in [38]). A similar result holds for ($P_{h}$).

Now let $z_{1}$ be a maximum slack in ($P$), then $(z_{1},1)$ is a slack in ($P_{h}$). Next, let $(z_{2},x_{0})$ be a maximum slack in ($P_{h}$). By the properties of the relative interior, and since $(z_{1},1)$ is a slack in ($P_{h}$), we have that $(z_{2},x_{0})-\epsilon(z_{1},1)$ is a slack in ($P_{h}$) for some $\epsilon>0.\,$ So $x_{0}>0$ must hold, and (after normalizing) we can assume $x_{0}=1.$ Hence $z_{2}$ is a slack in ($P$) and

will do. This completes the proof of the claim.

To proceed with the proof of the theorem, we note that the set of maximum slacks in ($P$) is a relatively open set, so by Theorem 18.2 in [38] it is contained in $\operatorname{ri}F,\,$ where $F$ is some face of $K.$ Therefore $\operatorname{dir}(z,K)=K+\operatorname{lin}F$ for any maximum slack $z$ (see e.g. Lemma 2.7 in [28]) so the sets $\operatorname{dir}(z,K)$ and $\tan(z,K)$ depend only on $F.$ Hence we are free to use any maximum slack of ($P$) in our proof, and we will use the particular maximum slack provided in the preceding Claim.

For convenience we define the linear map

which corresponds to the homogenized conic linear system ($P_{h}$).

We first note that (trivially)

Equations (1.4)–(1.6) imply that the same statement holds, if we replace the operator ${}^{\prime\prime}\operatorname{dir}^{\prime\prime}$ by ${}^{\prime\prime}\operatorname{cl}\operatorname{dir}^{\prime\prime},\,$ $``\tan^{\prime\prime},\,$ or ${}^{\prime\prime}\operatorname{ldir}^{\prime\prime}.$

Hence the following equations hold:

Consider now the following variants of conditions (1) and (2):

1. $(1^{\prime})$

   ${\cal R}(\mathcal{A}_{h})\cap\bigl{[}\operatorname{cl}\operatorname{dir}\bigl{(}(z,1),K\times\mathbb{R}_{+}\bigr{)}\setminus\operatorname{dir}\bigl{(}(z,1),K\times\mathbb{R}_{+}\bigr{)}\bigr{]}=\emptyset.$

2. $(2^{\prime})$

   There is $(u,u_{0})\in{\cal N}(A_{h}^{*})\cap(K\times\mathbb{R}_{+})^{*}\,$ strictly complementary to $(z,1)$ and

   $${\cal R}(\mathcal{A}_{h})\cap\bigl{[}\tan\bigl{(}(z,1),K\times\mathbb{R}_{+}\bigr{)}\setminus\operatorname{ldir}\bigl{(}(z,1),K\times\mathbb{R}_{+}\bigr{)}\bigr{]}=\emptyset.$$

Since $(z,1)$ is a maximum slack in ($P_{h}$), we have $(z,1)\in\operatorname{ri}\bigl{(}{\cal R}(\mathcal{A}_{h})\cap(K\times\mathbb{R}_{+})\bigr{)}.$ Hence by Lemma 1 with $C=K\times\mathbb{R}_{+},\,{\cal M}=\mathcal{A}_{h},\,w=(z,1)$ we find

and that equivalence holds when $K\times\mathbb{R}_{+}$ is nice.

We next note that by (2.9) condition $(1^{\prime})$ is equivalent to (1). Also, if $(u,u_{0})$ is as specified in $(2^{\prime}),$ then

and since both terms above are nonnegative, we must have $u_{0}=0.\,$ Thus using (2.10) we find that statement $(2^{\prime})$ is equivalent to condition (2) in Theorem 1. Thus we have

with equivalence holding when $K\times\mathbb{R}_{+}$ is nice. Finally, $K$ is nice if and only if $K\times\mathbb{R}_{+}$ is, thus invoking Lemma 2 completes the proof. ∎

We can easily modify the proof of Theorem 1 to show that conditions 1 and 2 suffice for ($P$) to be well behaved, even under a weaker condition than $K$ being nice: it is enough for $K^{*}+F^{\perp}$ to be closed, where $F$ is the smallest face of $K$ that contains $z.$ This more general version of Theorem 1 implies that Corollary 1 holds even if we do not assume that $K$ is nice – we refer the interested reader to version 3 of the paper on arxiv.org.

We now specialize the results of Section 2, and characterize when the semidefinite system ($P_{SD}$) is badly or well behaved. To this end, we consider the primal-dual pair of SDPs

where $A_{1},\dots,A_{m},B\in{\cal S}^{n},$ and $c_{1},\dots,c_{m}$ are scalars.

Specializing Definition 1 to the semidefinite system ($P_{SD}$), we find that i) a slack in ($P_{SD}$) is a matrix of the form $S=B-\sum_{i}x_{i}A_{i}\succeq 0,\,$ and ii) a maximum slack in ($P_{SD}$) is a maximum rank slack. We also note that the cone of positive semidefinite matrices is nice [16, 15, 29].

We make the following

We can easily satisfy Assumption 1, at least from a theoretical point of view, as follows. If $Z$ is any maximum rank slack in ($P_{SD}$), $Q$ is a matrix of suitably scaled eigenvectors of $Z,$ and we apply the rotation $Q^{T}()Q$ to all $A_{i}$ and $B,$ then the maximum rank slack in the rotated system is in the required form. (We do not make a claim about actually computing $Z$ or $Q$; we discuss this point more at the end of Section 4 ).

In the interest of the reader we first state and illustrate the main results, then prove them.

The $Z$ and $V$ matrices provide a certificate of the bad behavior of ($P_{SD}$).

We next characterize well behaved semidefinite systems:

We return to Examples 3–6 in Section 4. As we will see there, the bad or good behavior of semidefinite systems can be verified using only an elementary linear algebraic argument, without ever referring to Theorems 2 or 3. We will use Examples 3–6 as illustrations.

The reader may find it interesting to spot the $Z$ and $V$ excluded matrices in other pathological SDPs in the literature, e.g., in the instances in [6, 11, 3, 44, 41, 34, 43, 27].

Theorems 2 and 3 simply follow from Theorem 1 and from Lemma 3 below, which describes the set of feasible directions and related sets in the semidefinite cone:

∎

The proof of Lemma 3 is given in Appendix B.

To summarize, Theorems 2 and 3 are a ”combinatorial version” of Theorem 1.

We note that for semidefinite systems that are strictly feasible, a matrix similar to the $V$ matrix in Theorem 2 can make sure that the optimal primal-dual solution pair fails strict complementarity; see [48].

Although we focus on feasible systems, we obtain natural corollaries about weakly infeasible SDPs, a class of pathological infeasible SDPs. To describe the connection, note that the alternative system

gives a natural proof of infeasibility of ($P_{SD}$): if (3.24) is feasible, then ($P_{SD}$) is trivially infeasible. However, ($P_{SD}$) and (3.24) may both be infeasible, in which case we call the semidefinite system ($P_{SD}$) weakly infeasible.

As background on weakly infeasible SDPs, we mention that Waki [45] recently described a method for generating weakly infeasible SDPs based on Lasserre’s relaxation for polynomial optimization problems; Klep and Schweighofer [24] analyzed weakly infeasible SDPs using real algebraic geometry techniques; and Lourenco et al [26] proved that any weakly infeasible SDP with order $n$ matrices has a weakly infeasible subsystem with dimension at most $n-1.$

To apply our machinery to weakly infeasible SDPs, we homogenize ($P_{SD}$) to obtain the system