Research map / Learning theory and algorithms / Prediction and decision losses
Positive Concave Deep Equilibrium Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes positive concave deep equilibrium (pcDEQ) models that enforce nonnegative weights and concave activations so DEQ fixed points exist, are unique, and are found by plain fixed-point iteration with geometric convergence.
Problem
Standard deep equilibrium (DEQ) models lack formal guarantees of fixed-point existence, uniqueness, and convergence of the numerical fixed-point scheme, making them potentially unstable in practice.
Approach
Builds DEQ layers from positive concave (standard interference) mappings via nonlinear Perron-Frobenius theory: nonnegative weights plus NC/PC activations give a unique fixed point reachable by standard fixed-point iteration with geometric convergence, without monotone-operator constraints.
Key results
- On MNIST, pcDEQ-1-MC-ReLU6 reaches 99.3% test accuracy with 41K parameters, versus 99.0% for monDEQ-MT with 81K.
- On CIFAR-10 with data augmentation, pcDEQ-1-MC-Softsign reaches 89.0% with 661K parameters, comparable to monDEQ-MT* at 89.0% with 1M parameters.
- Fixed-point iteration for pcDEQ with a single convolutional layer (sigmoid) satisfies the stopping criterion in fewer than eight iterations, with no increase during training.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use pcDEQ-1-MC-ReLU6 instead of monDEQ-SC | Image classification model · MNIST test | accuracy: 99.1 -> 99.3 | Helped |
| Use pcDEQ-1-MC-ReLU6 instead of monDEQ-MT | Image classification model · SVHN test | accuracy: 92.4 -> 93.0 | Helped |
| Use pcDEQ-1-MC-ReLU6 instead of monDEQ-SC | Image classification model · CIFAR-10 test | accuracy: 74.0 -> 78.2 | Helped |
| Use pcDEQ-1-MC-ReLU6* instead of monDEQ-MT* | Image classification model with data augmentation · CIFAR-10 test | accuracy: 89.0 -> 89.2 | Helped |
| Add data augmentation to pcDEQ-1-MC-ReLU6 (ablation) | pcDEQ-1-MC-ReLU6 · CIFAR-10 test | accuracy: 78.2 -> 89.2 | Helped |
| Add data augmentation to pcDEQ-1-MC-Tanh (ablation) | pcDEQ-1-MC-Tanh · CIFAR-10 test | accuracy: 76.5 -> 88.5 | Helped |
| Add data augmentation to pcDEQ-1-MC-Softsign (ablation) | pcDEQ-1-MC-Softsign · CIFAR-10 test | accuracy: 77.1 -> 89.0 | Helped |
| Add data augmentation to pcDEQ-2-MC-Sigmoid (ablation) | pcDEQ-2-MC-Sigmoid · CIFAR-10 test | accuracy: 71.0 -> 85.6 | Helped |
| Use three convolutional layers rather than one with ReLU6 (ablation) | pcDEQ-1 · SVHN test | accuracy: 88.0 -> 93.0 | Helped |
| Use three convolutional layers rather than one with ReLU6 (ablation) | pcDEQ-1 · CIFAR-10 test | accuracy: 76.3 -> 78.2 | Helped |
| Use Softsign instead of ReLU6 in the three-convolution CIFAR-10 model with data augmentation (ablation) | pcDEQ-1-MC · CIFAR-10 test, data augmentation | accuracy: 89.2 vs 89.0 | No clear effect |
| Use pcDEQ-2-MC-Sigmoid instead of pcDEQ-1-MC-ReLU6 (ablation) | Three-convolution pcDEQ · CIFAR-10 test | accuracy: 78.2 -> 71.0 | Hurt |
Surprising findings
- For softsign, tanh and sigmoid networks the Lipschitz bound exceeds 1 for most of training, yet a unique fixed point still exists, showing Banach-type conditions are unnecessarily strict here.
- The pcDEQ-2-MC-Sigmoid variant gets only 71.0% on CIFAR-10 without augmentation, well below the other pcDEQ variants.
Limitations
- Non-negativity and concavity constraints restrict the architecture class.
- The Lipschitz-based bound for ReLU6 holds only empirically; no guarantee the bound stays below one under other settings.
- Results only on small image benchmarks (MNIST, SVHN, CIFAR-10) with no language modeling experiments.
Experimental setup
Datasets: MNIST SVHN CIFAR-10
Benchmarks: MNIST SVHN CIFAR-10
Metrics: test accuracy number of parameters fixed point iterations largest singular value (Lipschitz proxy)
Baselines: monDEQ NODE ANODE
Builds on: monDEQ Neural ODE ANODE Deep equilibrium models
Closest papers by meaning
- Consistency Deep Equilibrium Models (ICML 2026 · similarity 0.90)
- Separation and Bias of Deep Equilibrium Models on Expressivity and Learning Dynamics (NeurIPS 2024 · similarity 0.89)
- Lyapunov-Stable Deep Equilibrium Models (AAAI 2024 · similarity 0.89)
- Understanding Representation of Deep Equilibrium Models from Neural Collapse Perspective (NeurIPS 2024 · similarity 0.88)
- Subhomogeneous Deep Equilibrium Models (ICML 2024 · similarity 0.87)
- DeltaDEQ: Exploiting Heterogeneous Convergence for Accelerating Deep Equilibrium Iterations (NeurIPS 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.