Research map / Learning theory and algorithms / Kernels and regression theory
Subhomogeneous Deep Equilibrium Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes subhomogeneous deep equilibrium models (SubDEQ), proving existence and uniqueness of DEQ fixed points via subhomogeneous operators and nonlinear Perron-Frobenius theory without constraints on the weight matrices, and tests them on image classification and APPNP-style graph propagation.
Problem
Implicit-depth DEQ models often lack guarantees of fixed-point existence and uniqueness, which raises stability, performance and reproducibility concerns; existing monotone-operator fixes require restrictive weight parametrizations.
Approach
Define μ-subhomogeneous operators via Clarke's generalized Jacobian and use the Thompson projective metric (complete on the positive orthant) so that a positive subhomogeneous map with μ<1 is a contraction and has a unique fixed point; activations such as shifted tanh (tanh(z)+α) make the DEQ subhomogeneous with no assumption on W.
Key results
- On MNIST dense, SubDEQ (Tanh) reaches 1.92% error versus 2.736% for MonDEQ (Tanh) and 2.056% for MonDEQ (ReLU).
- On SVHN, SubDEQ (Normalized Tanh) achieves 9.36% error versus 11.04% for MonDEQ (ReLU) and 12.72% for the standard CNN.
- On CIFAR-10 convolutional, MonDEQ (ReLU) at 24.41% error outperforms SubDEQ variants (about 27.9–28.4%).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use SubDEQ (Tanh) instead of MonDEQ (ReLU) | MNIST (Dense) · MNIST (Dense) test | misclassification error: 2.056 -> 1.92 | Helped |
| Use SubDEQ (Tanh) instead of MonDEQ (ReLU) | MNIST (Convolutional) · MNIST (Convolutional) test | misclassification error: 0.654 -> 0.706 | Hurt |
| Use SubDEQ (Tanh) instead of MonDEQ (ReLU) | CIFAR-10 · CIFAR-10 test | misclassification error: 24.414 -> 27.946 | Hurt |
| Use SubDEQ (Normalized Tanh) instead of MonDEQ (ReLU) | SVHN · SVHN test | misclassification error: 11.0356 -> 9.3562 | Helped |
| Use SubDEQ (Normalized Tanh) instead of MonDEQ (ReLU) | TinyImageNet · TinyImageNet test | misclassification error: 85.49 -> 70.1791 | Helped |
| Replace APPNP with APPNP (Normalized Tanh) | Cora citation · Cora citation (5.2% labeled) test | accuracy: 80.2027 -> 73.3905 | Hurt |
| Replace APPNP with APPNP (Normalized Tanh) | Cora author · Cora author (5.2% labeled) test | accuracy: 70.834 -> 73.3437 | Helped |
| Replace APPNP with APPNP (Normalized Tanh) | CiteSeer · CiteSeer (4.2% labeled) test | accuracy: 62.8625 -> 62.66078 | Hurt |
| Replace APPNP with APPNP (Normalized Tanh) | DBLP · DBLP (4.0% labeled) test | accuracy: 88.8095 -> 89.4007 | Helped |
| Replace APPNP with APPNP (Normalized Tanh) | PubMed · PubMed (0.8% labeled) test | accuracy: 77.45168 -> 78.5827 | Helped |
| Use normalized Tanh shifted by 1.2 instead of 1.603 (ablation) | SubDEQ (Normalized Tanh) · MNIST (Dense) test | misclassification error: 2.088 -> 2.437 | Hurt |
| Use power-scaled Tanh instead of the 1.603-shifted Tanh (ablation) | SubDEQ (Normalized Tanh) · MNIST (Dense) test | misclassification error: 2.088 -> 2.568 | Hurt |
Surprising findings
- On CIFAR-10 the monotone MonDEQ (ReLU) beats the SubDEQ variants, so SubDEQ is not uniformly best.
- Adding the shift 1.603 needed for the normalized variant's uniqueness guarantee did not hurt accuracy and sometimes improved it.
Limitations
- Results are reported on relatively small benchmark setups with modest architectures, and SubDEQ does not win on every dataset (e.g., CIFAR-10 convolutional).
- Uniqueness theory requires positive activations/inputs or shifts (e.g., tanh+α), restricting the activation choices.
Experimental setup
Datasets: MNIST CIFAR-10 SVHN TinyImageNet Cora CiteSeer DBLP PubMed
Benchmarks: MNIST CIFAR-10 SVHN TinyImageNet Cora citation Cora author CiteSeer DBLP PubMed
Metrics: misclassification error accuracy relative residual
Baselines: MonDEQ (ReLU) MonDEQ (Tanh) nODE (ReLU) nODE (Tanh) Standard CNN Standard MLP APPNP
Builds on: monotone operator DEQ (MonDEQ) Neural ODE APPNP Deep equilibrium models
Closest papers by meaning
- Positive Concave Deep Equilibrium Models (ICML 2024 · similarity 0.87)
- Separation and Bias of Deep Equilibrium Models on Expressivity and Learning Dynamics (NeurIPS 2024 · similarity 0.86)
- Deep Equilibrium Models are Almost Equivalent to Not-so-deep Explicit Models for High-dimensional Gaussian Mixtures (ICML 2024 · similarity 0.86)
- DeltaDEQ: Exploiting Heterogeneous Convergence for Accelerating Deep Equilibrium Iterations (NeurIPS 2024 · similarity 0.86)
- Implicit regularization of deep residual networks towards neural ODEs (ICLR 2024 · similarity 0.86)
- Understanding Representation of Deep Equilibrium Models from Neural Collapse Perspective (NeurIPS 2024 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.