Research map / Optimization / Nonconvex and smooth optimization
Cauchy-Schwarz Regularizers
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces Cauchy-Schwarz (CS) regularizers, differentiable auto-scaling penalties derived from the CS inequality that promote discrete-valued vectors, eigenvectors, and orthogonal-column matrices, and applies them to sparse recovery and neural network weight quantization.
Problem
Designing differentiable regularizers that promote structured solution vectors (discrete-valued, eigenvectors, orthogonal matrices) without fixing scale or requiring lifting/matrix decompositions, for gradient-based solvers and large-scale problems.
Approach
Penalty equals the gap in the Cauchy-Schwarz inequality, ||g||^2||h||^2 - <g,h>^2, which is zero iff g and h are linearly dependent; choosing polynomial g and h yields regularizers that auto-adapt scale and are often free of spurious stationary points.
Key results
- Symmetric binary regularizer l_bin achieves higher recovery success probability than l-infinity minimization for all undersampling ratios in underdetermined systems with N=100.
- On ImageNet with ResNet-18, binary weights outperform SQ-BWN, BWN, HWGQ, PCCN and TWN but are 4.9% below ProxyBNN.
- On CIFAR-10 binary weights match LQ-Net and are 0.9% below DAQ and LCR-BNN.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use the CS symmetric-binary regularizer instead of ℓ∞-norm minimization | symmetric binary recovery from underdetermined linear systems · Underdetermined linear systems | success probability | Helped |
| Use the CS one-sided-binary regularizer instead of ℓ1-norm minimization | one-sided binary recovery from underdetermined linear systems · Underdetermined linear systems | success probability | Helped |
| Use the CS symmetric-ternary regularizer instead of ℓ1-norm minimization | symmetric ternary recovery from underdetermined linear systems · Underdetermined linear systems | success probability | Helped |
| Use the CS eigenvector regularizer instead of optimizing an additional eigenvalue parameter | eigenvector recovery from underdetermined linear systems · Underdetermined linear systems | success rate | Helped |
| Use the CS orthogonality regularizer for matrix recovery | recovery of matrices with orthogonal columns · Underdetermined linear systems | success rate | Helped |
| Use CS-regularized binary weight quantization instead of the best SOTA method | ImageNet ResNet-18 binary weight quantization · ImageNet | top-1 accuracy: 4.9% | Hurt |
| Use CS-regularized ternary weight quantization instead of the SOTA method | ImageNet ResNet-18 ternary weight quantization · ImageNet | top-1 accuracy: 2.8% | Hurt |
| Use CS-regularized binary weight quantization instead of DAQ or LCR-BNN | CIFAR-10 ResNet-20 binary weight quantization · CIFAR-10 | top-1 accuracy: 0.9% | Hurt |
| Use CS-regularized binary weight quantization instead of DoReFa-Net | CIFAR-10 ResNet-20 binary weight quantization · CIFAR-10 | top-1 accuracy | Helped |
| Use CS-regularized binary weight quantization instead of LQ-Net | CIFAR-10 ResNet-20 binary weight quantization · CIFAR-10 | top-1 accuracy | No clear effect |
| Use CS-regularized ternary weight quantization instead of DAQ or LCR-BNN | CIFAR-10 ResNet-20 ternary weight quantization · CIFAR-10 | top-1 accuracy: 0.2% | Hurt |
Surprising findings
- The eigenvector regularizer's spurious-stationary-point freeness could not be proven, so the authors refrain from claiming it.
- Ternarization regularizer involves eighth-order polynomials, causing poor convergence near the minimum.
Limitations
- CS regularizers are generally nonconvex, so restarts may be needed for global convergence.
- Higher-order polynomial regularizers (e.g., ternary, eighth order) can converge poorly near minima.
- Evaluated on only four example tasks.
Experimental setup
Datasets: ImageNet (ILSVRC12) CIFAR-10
Benchmarks: ImageNet (ILSVRC12) CIFAR-10
Metrics: top-1 accuracy probability of success for exact recovery
Baselines: l-infinity norm minimization l1 norm minimization TWN BWN DoReFa-Net LQ-Net ProxyBNN QIL
Builds on: Cauchy-Schwarz inequality Hölder inequality FISTA LQ-Net
Closest papers by meaning
- Implicit Regularization of Discrete Gradient Dynamics in Linear Neural Networks (NeurIPS 2019 · similarity 0.87)
- Fiedler Regularization: Learning Neural Networks with Graph Sparsity (ICML 2020 · similarity 0.86)
- Implicit Regularization in Matrix Factorization (NeurIPS 2017 · similarity 0.86)
- Connecting Optimization and Regularization Paths (NeurIPS 2018 · similarity 0.86)
- Implicit Regularization in Deep Matrix Factorization (NeurIPS 2019 · similarity 0.86)
- Weakly Convex Regularisers for Inverse Problems: Convergence of Critical Points and Primal-Dual Optimisation (ICML 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.