Research map / Learning theory and algorithms / Combinatorial and search optimization

Position: Adopt Constraints Over Fixed Penalties in Deep Learning

NeurIPS 2026 · Learning theory and algorithms › Combinatorial and search optimization · approach: Constrained neural network optimization · 1 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Position paper arguing that non-negotiable requirements in deep learning should be modeled as explicit constraints and solved with constrained methods (e.g., Lagrangian/primal-dual), rather than fixed weighted-sum penalties.

Problem

Deep learning problems with non-negotiable requirements are often handled by fixed weighted-sum penalties, which in non-convex settings may not recover the constrained solution, weaken hard requirements into soft trade-offs, and require costly coefficient tuning.

Approach

Argues that the constrained formulation should be the starting point; uses a concave 1D counterexample to show no fixed penalty coefficient recovers the constrained optimum, and advocates primal-dual/augmented Lagrangian methods chosen by problem structure and scale.

Key results

• On the concave 1D example, fixed penalization returns only endpoints for every coefficient c, while the augmented Lagrangian recovers x*=arcsin(epsilon) across eight constraint levels.
• In the rate-constrained task, only one penalty coefficient (c=2.15) yields a non-collapsed solution, while the Lagrangian reaches the 70% target over a broad range of dual step-sizes.
• Constrained training adds only about 5.2% (classification) and 1.6% (sparsity) wall-clock overhead relative to penalized training.

Surprising findings

• Even setting the penalty coefficient to the optimal Lagrange multiplier, fixed penalization fails to recover the constrained minimizer in the non-convex concave example.
• Most penalty coefficients in the rate-constrained experiment collapse to either 50% or 100% class-0 predictions.

Limitations

• Lagrangian use is illustrative; the paper argues the appropriate method depends on problem structure and scale.
• Constrained methods can be harder to optimize and deploy, and constraint satisfaction may not generalize beyond training data.
• Experiments are small toy or MNIST-scale settings; overhead measurements are limited.

Experimental setup

Datasets: MNIST synthetic 2D Gaussian mixture

Benchmarks: concave 1D toy problem MNIST sparsity-constrained training rate-constrained linear classification

Metrics: objective value f constraint satisfaction (density, class-0 percentage) accuracy wall-clock time

Baselines: fixed weighted-sum penalization gradient descent-ascent Lagrangian

Builds on: Lagrangian methods augmented Lagrangian Cooper Hard Concrete L0 regularization nuPI

Closest papers by meaning

• Near-Optimal Solutions of Constrained Learning Problems (ICLR 2024 · similarity 0.86)
• Enforcing Hard Linear Constraints in Deep Learning Models with Decision Rules (NeurIPS 2025 · similarity 0.83)
• DeepSaDe: Learning Neural Networks That Guarantee Domain Constraint Satisfaction (AAAI 2024 · similarity 0.83)
• Improving Deep Learning Optimization through Constrained Parameter Regularization (NeurIPS 2024 · similarity 0.83)
• Deep Weight Factorization: Sparse Learning Through the Lens of Artificial Symmetries (ICLR 2025 · similarity 0.82)
• Beyond $\tilde{O}(\sqrt{T})$ Constraint Violation for Online Convex Optimization with Adversarial Constraints (NeurIPS 2025 · similarity 0.82)