Research map / Architectures and efficiency / RNNs and normalization
Separation and Bias of Deep Equilibrium Models on Expressivity and Learning Dynamics
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Theoretical separations between deep equilibrium models (DEQ) and feedforward networks (FNN) in expressivity (linear regions, steep functions) and an implicit-bias analysis of gradient flow/descent for a diagonal linear DEQ, supported by small experiments.
Problem
Understanding when and why DEQs have provable advantages over explicit feedforward networks, in terms of expressive power and learning dynamics (implicit bias), which prior work (well-posedness, NTK analyses) left open.
Approach
Shows a width-m ReLU DEQ can realize 2^m linear regions, giving a separation from depth-O(m^α) FNNs; constructs a steep target function where DEQ approximates in L∞ with polynomial size while bounded-depth FNNs need exponentially large weights; and characterizes the implicit regularization of gradient flow in a diagonal linear DEQ as minimizing a sum of convex penalties q_i(β_i)=½β_i²+β_i(0)^{-3}β
Key results
- A width-m ReLU DEQ can have 2^m linear regions, while depth-O(m^α) FNNs of bounded width cannot match it unless width is exp(Ω(m^{1-α})).
- For the steep function g(x) with δ=2^{-d}, a DEQ with width 5/ε and weights bounded by 2/ε approximates to ε in L∞, whereas bounded-depth FNNs need weights of order 2^{Ω(d)}.
- For a diagonal linear DEQ with bias, the GOTU error on Proposition 2 setting is bounded by a value below 4 f̂({k})², improving on the diagonal linear network bound of 4 f̂({k})²+O(ε).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use a width-m ReLU DEQ instead of a not-so-deep FNN | ReLU networks · Piecewise-linear functions | linear regions: 2^m linear regions | Helped |
| Approximate the target with a width-m DEQ rather than a depth-L FNN of width at most 2m^(1−α)−1 | ReLU networks · Constructed separation function on [0,1]^d | L1 approximation error: FNN error ≥ 1/16 | Helped |
| Represent the steep target using a DEQ | Approximation of g(x) on [0,1]^d · Steep function g(x) | L∞ approximation error: ≤ ε | Helped |
| Use a diagonal linear DEQ with bias rather than a diagonal linear network | OOD learning · GOTU unseen domain | GOTU error: ≤ 4( f̂({k}) − (4 + 3f̂({k}))⁻¹ᐟ³ )² < 4f̂({k})²; baseline: 4f̂({k})² + O(ε) | Helped |
| Use DEQ instead of FNN for the saw-tooth function experiments | FNN and DEQ · Saw-tooth functions with 2⁵ and 2¹⁰ linear regions | test loss | Helped |
| Use DEQ instead of FNN with a similar parameter count | FNN and DEQ · Quadratic-optimization solution g(x), δ = 2⁻¹⁰ and 2⁻²⁰ | test loss | Helped |
| Use DEQ instead of FNN on the unseen-domain Boolean-function tasks | FNN and DEQ · OOD functions f1 and f2 | GOTU error | Helped |
| Train with diagonal linear DEQ dynamics | Diagonal linear DEQ · Overparameterized interpolation | feature density | Helped |
Surprising findings
- Linear regions of a width-m DEQ reach 2^m, a bound that is shown tight, while FNN bounds on linear regions are not known to be achievable.
Limitations
- Bias analysis restricted to a simplified diagonal linear DEQ with gradient flow/descent (not stochastic methods)
- Separation results are existence/worst-case statements for specific constructed target functions
- Conjecture about high-frequency learning is not fully proven
Experimental setup
Datasets: synthetic sawtooth functions synthetic steep function g(x) synthetic boolean functions on {-1,1}^d
Metrics: test loss GOTU error L-infinity approximation error
Baselines: feedforward neural network two-layer linear network diagonal linear network
Builds on: deep equilibrium model Telgarsky depth separation implicit bias of diagonal linear networks
Closest papers by meaning
- Understanding Representation of Deep Equilibrium Models from Neural Collapse Perspective (NeurIPS 2024 · similarity 0.89)
- Positive Concave Deep Equilibrium Models (ICML 2024 · similarity 0.89)
- Deep Equilibrium Models are Almost Equivalent to Not-so-deep Explicit Models for High-dimensional Gaussian Mixtures (ICML 2024 · similarity 0.89)
- Lyapunov-Stable Deep Equilibrium Models (AAAI 2024 · similarity 0.87)
- DeltaDEQ: Exploiting Heterogeneous Convergence for Accelerating Deep Equilibrium Iterations (NeurIPS 2024 · similarity 0.86)
- Subhomogeneous Deep Equilibrium Models (ICML 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.