Research map / Learning theory and algorithms / Kernels and regression theory
Deep Equilibrium Models are Almost Equivalent to Not-so-deep Explicit Models for High-dimensional Gaussian Mixtures
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Uses random matrix theory on high-dimensional Gaussian mixtures to show DEQ conjugate kernel and NTK spectra depend only on four nonlinear equations, and designs shallow explicit networks (H-Tanh, L-ReLU activations) that match DEQ CK/NTK and performance.
Problem
Lack of theoretical understanding of how implicit deep equilibrium models (DEQs) relate to explicit neural networks in terms of their conjugate kernel and NTK spectra, and whether an equivalent explicit network exists for a given DEQ.
Approach
Derives high-dimensional deterministic equivalents for the implicit DEQ conjugate kernel and NTK under Gaussian mixture data, showing they depend on the activation and variance only through four scalars; matches these with a shallow explicit network whose activation is designed to reproduce the same coefficients.
Key results
- Implicit CK and NTK are approximated by explicit deterministic equivalents with spectral-norm error O(n^{-1/2}), depending on activation and variance only via four scalar equations.
- Relative spectral-norm error between the implicit CK and its approximation decreases as n and p grow for ReLU, Tanh, Swish and Leaky-ReLU DEQs.
- On MNIST, Fashion-MNIST and CIFAR-10, the L-ReLU-ENN closely matches ReLU-DEQ accuracy as width grows, while standard ReLU-ENN shows a noticeable gap.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Design shallow explicit-network activations to match the DEQ CK | Tanh-DEQ and H-Tanh-ENN · GMM and MNIST | CK spectral norm difference: O(n^{-1/2}) | Helped |
| Design shallow explicit-network activations to match the DEQ CK | ReLU-DEQ and L-ReLU-ENN · GMM and MNIST | CK spectral norm difference: O(n^{-1/2}) | Helped |
| Use the theoretically designed Hard-Tanh-type activation instead of Tanh | Tanh-DEQ-matched explicit network · MNIST, Fashion-MNIST, and CIFAR-10 | classification accuracy | No clear effect |
| Use the theoretically designed Leaky-ReLU-type activation instead of ReLU | ReLU-DEQ-matched explicit network · MNIST, Fashion-MNIST, and CIFAR-10 | classification accuracy | Helped |
| Replace the designed Hard-Tanh-type activation with Tanh | Tanh-DEQ-matched explicit network · MNIST, Fashion-MNIST, and CIFAR-10 | classification accuracy | Hurt |
| Replace the designed Leaky-ReLU-type activation with ReLU | ReLU-DEQ-matched explicit network · MNIST, Fashion-MNIST, and CIFAR-10 | classification accuracy | Hurt |
| Increase explicit-network width | L-ReLU-ENN · MNIST, Fashion-MNIST, and CIFAR-10 | classification accuracy | Helped |
| Increase sample size | Implicit-CK high-dimensional approximation · Binary GMM | relative CK spectral norm error | Helped |
Surprising findings
- A single-hidden-layer explicit network cannot match every DEQ CK; some DEQs require at least two hidden layers.
- The high-dimensional approximations derived for Gaussian mixtures also match behavior on real MNIST data.
Limitations
- Theory is derived for high-dimensional Gaussian mixture data; real-data agreement is empirical and conjectured to be universal.
- Results hold in the large-width and large n,p limit, and require assumptions such as centered activations and contractive variance conditions.
Experimental setup
Datasets: MNIST Fashion-MNIST CIFAR-10 synthetic two-class Gaussian mixture
Benchmarks: MNIST Fashion-MNIST CIFAR-10 synthetic Gaussian mixture
Metrics: relative spectral norm error classification accuracy training and inference time
Baselines: Tanh-ENN ReLU-ENN ReLU-DEQ Tanh-DEQ
Builds on: deep equilibrium models neural tangent kernel random matrix theory for kernel analysis of neural networks
Closest papers by meaning
- Separation and Bias of Deep Equilibrium Models on Expressivity and Learning Dynamics (NeurIPS 2024 · similarity 0.89)
- Eigen Analysis of Conjugate Kernel and Neural Tangent Kernel (ICML 2025 · similarity 0.88)
- Subhomogeneous Deep Equilibrium Models (ICML 2024 · similarity 0.86)
- Understanding Representation of Deep Equilibrium Models from Neural Collapse Perspective (NeurIPS 2024 · similarity 0.86)
- Positive Concave Deep Equilibrium Models (ICML 2024 · similarity 0.85)
- DeltaDEQ: Exploiting Heterogeneous Convergence for Accelerating Deep Equilibrium Iterations (NeurIPS 2024 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.