Position: Adopt Constraints Over Penalties
in Deep Learning

A popular approach to tackling Eq.˜1 in deep learning applications is the penalized approach—also known as a weighted-sum scalarization of multi-objective problems (Ehrgott, 2005, §3)—which minimizes a linear combination of the objective and the constraints:

where $\bm{c}_{\bm{g}}\succeq\bm{0}$ and $\bm{c}_{\bm{h}}$ are coefficients that encourage constraint satisfaction.

In deep learning applications, the popularity of the penalized approach likely stems from several factors: \raisebox{-.7pt}{{\hspace{-0.8mm} \small1}}⃝ it reformulates constrained problems as unconstrained ones, allowing the use of familiar gradient-based optimization tools without requiring specialized knowledge of constrained optimization; \raisebox{-.7pt}{{\hspace{-0.8mm} \small2}}⃝ it integrates smoothly with existing training protocols—such as Adam (Kingma and Ba, 2015) or learning rate schedulers—which are often highly specialized for deep neural network training (Dahl et al., 2023); and \raisebox{-.7pt}{{\hspace{-0.8mm} \small3}}⃝ it can be applied to problems where both the objective and constraint functions are differentiable, without relying on assumptions like convexity, making it well-suited to modern deep learning tasks.

Moreover, the penalized approach is computationally efficient in the context of automatic differentiation. The gradient of $\mathcal{P}$ is a linear combination of the gradients of $f$, $\bm{g}$ and $\bm{h}$, and can be computed without storing each gradient separately. Since constraints often share intermediate computations with the objective—such as forward passes through the model—additional forward calculations and the backward pass can be executed with minimal overhead compared to optimizing a single function.

In the convex setting, the penalized approach is justified in the following sense:

Consequently, when \raisebox{-.7pt}{{\hspace{-0.8mm} \small1}}⃝ $f$, $\bm{g}$, and $\bm{h}$ are convex; \raisebox{-.7pt}{{\hspace{-0.8mm} \small2}}⃝ the constrained minimizer of Eq.˜1 is regular; and \raisebox{-.7pt}{{\hspace{-0.8mm} \small3}}⃝ reliable estimates of its corresponding optimal multipliers $\bm{\lambda}^{*}$ and $\bm{\mu}^{*}$ are available, minimizing $\mathcal{P}$ is a principled approach to solving Eq.˜1.

Furthermore, under these convexity assumptions, varying the weights $(\bm{c}_{\bm{g}},\bm{c}_{\bm{h}})$ allows the penalized formulation to recover every attainable (weakly) Pareto-optimal⁴⁴4For vector-valued functions $f_{i}:\mathbb{R}^{d}\to\mathbb{R}$, a point $x^{\star}$ is weakly Pareto-optimal if there is no other feasible point $x$ such that $f_{i}(x)<f_{i}(x^{\star})$ for every index $i$. point $(f^{\star},\bm{\epsilon}_{\bm{g}},\bm{\epsilon}_{\bm{h}})$ in the image of the map $\bm{x}\mapsto(f(\bm{x}),\bm{g}(\bm{x}),\bm{h}(\bm{x}))$ (Miettinen, 1999, Thm. 3.1.4, §3.1). Hence, with a suitable choice of coefficients, the penalized approach can recover a constrained minimizer of Eq.˜1 for all attainable constraint levels. In this sense, the penalized and constrained formulations are equivalent for convex problems.

However, while effective in the convex setting and applicable to any differentiable constrained problem, the penalized approach is ill-suited. First, Prop.˜1 no longer holds when $f$, $\bm{g}$, or $\bm{h}$ are non-convex: as discussed in §3.1, certain realizable values of $\bm{g}$ and $\bm{h}$ may be unattainable by minimizing the penalized objective $\mathcal{P}$, regardless of the choice of penalty coefficients ($\bm{c}_{\bm{g}}$, $\bm{c}_{\bm{h}}$). Therefore, constrained minimizers for certain attainable constraint levels $(\bm{\epsilon}_{\bm{g}},\bm{\epsilon}_{\bm{h}})$ may not be recoverable using the penalized approach—even though proper constrained methods can recover them (see §4.2).

Moreover, the success of the penalized approach relies on having accurate estimates of the optimal Lagrange multipliers, such that $\bm{c}_{\bm{g}}\approx\bm{\lambda}^{*}$ and $\bm{c}_{\bm{h}}\approx\bm{\mu}^{*}$. In practice, however—and especially in non-convex problems—these multipliers are unknown. As a result, practitioners often resort to a costly trial-and-error process to tune the penalty coefficients (see §3.2).

In non-convex settings, the guarantees in Prop.˜1 no longer hold. The penalized approach may then fail to recover certain achievable constraint levels—regardless of the choice of penalty coefficients—resulting in an ill-posed formulation that cannot recover constrained minimizers of Eq.˜1.

Specifically, while (approximately) feasible solutions for inequality-constrained problems can always be recovered by choosing a sufficiently large penalty coefficient, recovering constrained-optimal solutions may be impossible—especially when these involve active constraints $\bm{g}(\bm{x})=\bm{\epsilon}_{\bm{g}}$, which is often the case.⁵⁵5Under inequality constraints, optima often lie on the boundary: reducing $\bm{g}$ below $\bm{\epsilon}_{\bm{g}}$ typically worsens $f$. The situation is even more problematic for equality constraints: if the target value $\bm{h}(\bm{x})=\bm{\epsilon}_{\bm{h}}$ is unattainable, the penalized approach may fail to produce any feasible solution at all.

In extreme cases, the penalized objective can completely disregard certain functions and focus solely on minimizing others. This failure mode is illustrated by the two-dimensional inequality-constrained problem in Example˜1, originally presented in an insightful blog post pair by Degrave and Korshunova (2021a, b). We formalize their empirical observation of the penalized method’s failure in Prop.˜3.

Prop.˜3 shows that for any choice of penalty coefficient, the penalized formulation of Example˜1 either recovers the unconstrained minimizer of $f$ by effectively ignoring the constraint $g$—resulting in an infeasible solution—or prioritizes minimizing $g$, yielding a feasible point with $g(x^{*},y^{*})=0<\epsilon$, but at the cost of significantly suboptimal performance on $f$. Notably, even when the penalty coefficient is set to the optimal Lagrange multiplier of the constrained problem ($c=\epsilon/\sqrt{1-\epsilon^{2}}$), the penalized formulation still fails to recover the constrained minimizer at $(x^{*},y^{*})=(\arcsin(\epsilon),0)$.

The penalized approach fails to solve Example˜1 because its objective $\mathcal{P}_{c}$ is concave in $x$ for any coefficient choice, pushing minimizers to the domain boundaries. This issue extends to general non-convex problems: if a (local) constrained optimum lies within a non-convex region of the penalized function $\mathcal{P}$, it becomes unreachable, as it does not minimize $\mathcal{P}$. Moreover, verifying whether $\mathcal{P}$ is convex around a solution often requires knowing the solution—i.e., solving the problem first.

A note on multi-objective problems. The exploration failure of the penalized approach carries over to multi-objective settings, where the goal is to minimize several objectives simultaneously. Its inability to capture the full range of optimal trade-offs can limit exploration of the Pareto front (Ehrgott, 2005, §4). As with constrained problems, we emphasize the importance of using specialized techniques for non-convex multi-objective optimization rather than relying on the penalized approach.⁶⁶6The constrained approach is a valid method for multi-objective optimization; see Ehrgott (2005, §4.1).

Even when the penalized formulation is principled—i.e., when Prop.˜1 holds—selecting appropriate penalty coefficients remains a costly trial-and-error process. The penalized problem often must be solved repeatedly with different coefficient values before arriving at a solution that satisfies the constraints and delivers good performance. This iterative re-training wastes both computational resources and development time, placing a significant burden on the community.

The unnecessarily lengthy process of tuning penalty coefficients is illustrated in Fig.˜1, where the goal is to train a neural network under a constraint enforcing at least $50\%$ sparsity. Using bisection search—which leverages the monotonic relationship between the coefficient and the resulting constraint value—the penalized problem must be solved five times before finding a satisfactory solution: one that is feasible but not overly sparse, as excessive sparsity would degrade performance. Even with this simple tuning method, the example demonstrates the process’s inefficiency. In contrast, a proper constrained optimization method can solve the problem in one shot, requiring only a single model training run (Gallego-Posada et al., 2022).

Several factors make coefficient tuning challenging: \raisebox{-.7pt}{{\hspace{-0.8mm} \small1}}⃝ The relationship between a given coefficient and the resulting values of $f$, $\bm{g}$, and $\bm{h}$ is often highly non-linear, requiring specialized hyperparameter tuning techniques or even manual intervention; \raisebox{-.7pt}{{\hspace{-0.8mm} \small2}}⃝ Penalty coefficients lack intuitive semantics, making them difficult to interpret; and \raisebox{-.7pt}{{\hspace{-0.8mm} \small3}}⃝ There is no general principled strategy for initializing them.

In contrast, constraint levels $(\bm{\epsilon}_{\bm{g}},\bm{\epsilon}_{\bm{h}})$ in constrained formulations \raisebox{-.7pt}{{\hspace{-0.8mm} \small1}}⃝ map directly to constraint values and \raisebox{-.7pt}{{\hspace{-0.8mm} \small2}}⃝ are interpretable, since their units match those of the constraints. In constrained approaches that learn the “coefficients”—e.g., the multipliers in the Lagrangian approach—\raisebox{-.7pt}{{\hspace{-0.8mm} \small3}}⃝ initialization is less critical, as they adapt during optimization and are thus typically set to zero for simplicity.

Moreover, tuning penalty coefficients becomes increasingly impractical in the presence of multiple constraints, and is further complicated by the fact that coefficient choices rarely transfer across tasks.

Multiple constraints. Tuning penalty coefficients becomes substantially more difficult in the presence of multiple constraints. Unlike the single-constraint case, multiple coefficients must be tuned jointly, as each one influences the satisfaction of all constraints. Moreover, while the monotonic relationship between coefficients and constraint values is straightforward to exploit in the single-constraint setting (e.g., as demonstrated in Fig.˜1), extending this to the multi-constraint setting is challenging. This interdependence significantly increases the complexity of the tuning process, often rendering it impractical and potentially leading practitioners to accept suboptimal—or even infeasible—solutions.

Penalty coefficients rarely transfer across settings. The same coefficient can yield different outcomes across architectures and datasets, due to changes in the landscape of the penalized objective. Moreover, adding new constraints to an existing problem typically requires re-tuning all coefficients, as feasibility may depend on rebalancing them to account for both existing and new constraints.

In contrast, the hyperparameters of the constrained approach—the constraint levels $\bm{\epsilon}_{\bm{g}}$ and $\bm{\epsilon}_{\bm{h}}$—are agnostic to the number of constraints, their parameterization, or their relationship to model inputs. While some constrained methods may still require hyperparameter tuning when tasks change or new constraints are added, these typically transfer more reliably than penalty coefficients (see §4.2).

The Lagrangian approach (Arrow et al., 1958) amounts to finding stationary points of the Lagrangian, which may correspond to solutions of the original constrained optimization problem (see §2):

This reformulates the constrained problem as an effectively unconstrained⁷⁷7Given that $\bm{\lambda}\succeq\bm{0}$ can be easily handled via projections. optimization problem. Notably, the Lagrangian is always linear—and therefore concave—in the multipliers.

A simple algorithm for finding stationary points of $\mathcal{L}$ is alternating gradient descent–ascent (GDA):

where $[\,\cdot\,]_{+}$ denotes a projection onto $\mathbb{R}_{\succeq\bm{0}}^{m}$ to enforce $\bm{\lambda}\succeq\bm{0}$, $[\,\cdot\,]_{\mathcal{X}}$ denotes a projection onto $\mathcal{X}$, and $\eta_{\{\text{primal},\text{dual}\}}$ are step-sizes. The multipliers are typically initialized as $\bm{\lambda}_{0}=\bm{0}$ and $\bm{\mu}_{0}=\bm{0}$. We favor alternating updates over simultaneous ones due to their enhanced convergence guarantees under a strongly convex $\mathcal{L}$ (Zhang et al., 2022), without incurring additional computational cost on Lagrangian games (Sohrabi et al., 2024).

Note that the update with respect to $\bm{x}$ in Eq.˜8 matches that of (projected) gradient descent on the penalized objective, with $\bm{c}_{\bm{g}}=\bm{\lambda}_{t+1}$ and $\bm{c}_{\bm{h}}=\bm{\mu}_{t+1}$. However, unlike penalty coefficients, which require manual tuning, the multipliers are instead optimized to balance feasibility and optimality.

The Lagrangian approach has been successfully applied to a wide range of constrained deep learning problems (Cotter et al., 2019b; Hashemizadeh et al., 2024; Gallego-Posada et al., 2022; Robey et al., 2021; Stooke et al., 2020; Elenter et al., 2022; Hounie et al., 2023b; Dai et al., 2024; Hounie et al., 2023a; Ramirez et al., 2025). Beyond this effectiveness, it is also viable in practice, as discussed below.

Dynamics of GDA. Consider a single inequality constraint $g(\bm{x})\leq 0$. The corresponding dual variable $\lambda$ is updated based on the constraint violation: when $g(\bm{x})>0$, $\lambda$ increases by $\eta_{\text{dual}}\,g(\bm{x})$. Repeated violations drive $\lambda$ larger, which shifts the primal update toward reducing $g(\bm{x})$ and promoting feasibility. Conversely, when $g(\bm{x})<0$, $\lambda$ decreases. In turn, a smaller multiplier shifts focus away from constraint satisfaction and toward minimizing the objective. When $\lambda=0$, the constraint is ignored. For instance, consider a classification problem with a sparsity constraint. When the number of non-zero parameters exceeds the constraint level, the multiplier increases, pushing the model to be sparser. For an equality constraint $h(\bm{x})=\epsilon$, the multiplier $\mu$ increases when $h(\bm{x})>\epsilon$. Conversely, it decreases—becoming negative—when $h(\bm{x})<\epsilon$. These dynamics encourage $h$ to approach $\epsilon$.

Computational cost. Applying (mini-batch) GDA to the Lagrangian is as efficient as applying (mini-batch) GD to the penalized objective $\mathcal{P}$, since the primal updates in both cases follow a linear combination of the objective and constraint gradients. The only overhead is updating the multipliers, which is negligible when the number of constraints is much smaller than $\text{dim}(\mathcal{X})$.

Tuning $\eta_{\text{dual}}$. A larger dual step-size makes the multipliers respond more aggressively to constraint violations: if $\eta_{\text{dual}}$ is too small, constraints may remain unsatisfied within the optimization budget; if too large, rapidly growing multipliers can overshoot their targets $\bm{\epsilon}_{\bm{g}}$ and $\bm{\epsilon}_{\bm{h}}$. In practice, however, tuning $\eta_{\text{dual}}$ is significantly easier than tuning penalty coefficients—a point we elaborate on in §4.2.

Implementation. The Lagrangian approach is straightforward to implement in popular deep learning frameworks such as PyTorch (Paszke et al., 2019) and JAX (Bradbury et al., 2018). Open-source libraries like Cooper (Gallego-Posada et al., 2025) for PyTorch and TFCO (Cotter et al., 2019c) for TensorFlow (Abadi et al., 2016) offer ready-to-use implementations, along with additional features such as the Augmented Lagrangian (Bertsekas, 1975), Proxy Constraints (Cotter et al., 2019b), Multiplier Models (Narasimhan et al., 2020), and $\nu$PI (Sohrabi et al., 2024).

Convergence. If $\mathcal{L}$ is strongly convex in $\bm{x}$, Eq.˜7 becomes a strongly convex–concave min–max problem, for which alternating GDA enjoys local linear convergence (Zhang et al., 2022). Moreover, simultaneous GDA converges to local min–max points in non-convex–concave settings, provided the maximization domain is compact (Lin et al., 2025). While the domains of $\bm{\lambda}$ and $\bm{\mu}$ are unbounded, if the optimal multipliers are finite, restricting optimization to a bounded region containing them allows these guarantees to hold.

Generalization. Chamon and Ribeiro (2020) develop a PAC learning framework for constrained learning. They argue that hypothesis classes that are PAC-learnable in the unconstrained setting may remain learnable under statistical constraints, probably approximately satisfying them on unseen data.

This section elaborates on why the simple Lagrangian approach is, in many respects, preferable to the penalized formulation. In short, transitioning to the Lagrangian approach—which can be seen as a principled alternative to manually tuning penalty coefficients by instead increasing them when constraints are violated and decreasing them when satisfied—yields an algorithm that: \raisebox{-.7pt}{{\hspace{-0.8mm} \small1}}⃝ is explicitly designed to solve the constrained problem, and is therefore expected to do so; and \raisebox{-.7pt}{{\hspace{-0.8mm} \small2}}⃝ requires substantially less manual hyperparameter tuning. As discussed in §4.3, it also offers several additional benefits, many of which remain relatively underexplored in the deep learning literature.

The Lagrangian approach is constrained by design. Lagrangian optimization continues until a feasible solution is found. In contrast, the penalized approach may converge while still producing an infeasible solution. This constrained-by-design philosophy also introduces a notion of accountability: by explicitly specifying acceptable values for $\bm{g}$ and $\bm{h}$ in the formulation, we adopt a more principled framework for both solving the problem and evaluating success—where infeasibility is not acceptable.

The Lagrangian approach can solve non-convex problems. As shown in Table˜1 (App. C.1), the Lagrangian approach successfully recovers the correct solution to Example˜1. Note that this requires either a specialized dual optimizer such as $\nu$PI (Sohrabi et al., 2024), or the Augmented Lagrangian method; standard gradient descent-ascent fails in this setting due to the concavity of the Lagrangian with respect to $x$. $\nu$PI overcomes this by dampening the oscillations that arise with GDA, while the Augmented Lagrangian method convexifies the problem—both enabling convergence to the constrained optimum (see App. D for details). Hyperparameter choices for Table˜1 are provided in App. B.1.

Hyperparameter tuning. The Lagrangian approach eliminates the need to tune penalty coefficients, but introduces a complementary challenge: selecting the dual step-size $\eta_{\text{dual}}$. However, this tuning is generally more forgiving—whereas typically only a single choice of penalty coefficient yields a satisfactory solution, a broad range of dual step-sizes often leads to feasibility, sometimes spanning several orders of magnitude. This is because the Lagrangian approach always pursues constraint satisfaction—regardless of how aggressively, as determined by the step-size—unlike the penalized approach, which is agnostic to the constraint levels. For this reason, we argue that this relative ease of tuning likely extends to other constrained optimization methods beyond the Lagrangian approach.

Figure˜2 illustrates this on a rate-constrained binary classification task (Cotter et al., 2019b), where 70% of predictions are required to belong to class “0”. Since the data is balanced, the optimal constrained solution should assign 70% of predictions to class 0—this is achieved with an optimal Lagrange multiplier of $\lambda^{*}=1.9975$, as found experimentally via the Lagrangian approach. The penalized formulation is infeasible for all $c<2$; for $c>2$, it satisfies the constraint but collapses to predicting a single class, degrading performance. Only at $c=2$ does it yield a reasonable prediction rate of 76%. In contrast, the Lagrangian approach with proxy constraints (Cotter et al., 2019b) consistently recovers the correct solution over a wide range of dual step-sizes by automatically learning the appropriate multiplier (see App. B.3).

Recent advances in constrained learning have equipped the Lagrangian approach with new capabilities that further strengthen its appeal in deep learning applications.

Non-differentiable constraints. Cotter et al. (2019b) show that the Lagrangian approach can be extended to satisfy non-differentiable constraints by using a differentiable surrogate during the primal update—where constraint gradients are required—while relying on the original, non-differentiable constraint when updating the multipliers. This “proxy constraints” approach ensures that constraint enforcement is driven by the true quantity of interest, independent of the choice of surrogate.

This capability is particularly relevant in machine learning, where optimization often relies on differentiable surrogates that do not match the true objective. While unconstrained empirical risk minimization offers no direct way to optimize for accuracy—e.g., optimizing cross-entropy instead—accuracy-based constraints can be enforced via the Lagrangian formulation with proxy constraints (Cotter et al., 2019b; Hashemizadeh et al., 2024).

Scaling the number of constraints. Each additional constraint introduces one more Lagrange multiplier—a single scalar. While this already scales efficiently, especially given the Lagrangian approach’s greater robustness to hyperparameters compared to the penalized formulation (§4.2), scalability can be further improved using parametric Lagrange multiplier models (Narasimhan et al., 2020). These models use a fixed set of parameters to represent all multipliers, regardless of the number of constraints, and can thus handle even problems with infinitely many constraints—unlike the penalized approach.

Sensitivity analysis via the multipliers. A standard result states that the optimal Lagrange multipliers $\bm{\lambda}^{*}$ and $\bm{\mu}^{*}$ capture how much the objective would improve if a constraint were relaxed (Boyd and Vandenberghe, 2004, §5.6). This can be exploited to design algorithms that trade off constraint tightness for performance (Hounie et al., 2024), making the Lagrangian approach well suited to settings with flexible or uncertain constraint levels.

Efficient dual variance reduction. Variance reduction techniques (Gower et al., 2020), popular for accelerating convergence in unconstrained optimization, have seen limited adoption in deep learning due to poor scalability—e.g., SAGA (Defazio et al., 2014) requires storing gradients for each datapoint. In contrast, they are inexpensive to apply to Lagrange multiplier updates, as the dual gradient is simply the constraint value, and have been shown to enhance convergence for problems involving stochastic constraints (Hashemizadeh et al., 2024).

Hyperparameter scaling. We hypothesize that, for a given task, the choice of dual step-sizes transfers well across model architectures and datasets. This would make hyperparameter scaling in the constrained approach significantly easier—allowing practitioners to tune parameters on smaller models and reuse them as effective starting points for larger-scale training, saving time and resources.