Research map / Architectures and efficiency / RNNs and normalization
DeltaDEQ: Exploiting Heterogeneous Convergence for Accelerating Deep Equilibrium Iterations
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes DeltaDEQ, which exploits heterogeneous convergence in deep equilibrium models by caching linear-operator results and propagating only thresholded state deltas, cutting forward-pass FLOPs on implicit neural representation and optical flow tasks.
Problem
Deep equilibrium models and other implicit networks require many fixed-point iterations at inference, which makes them computationally expensive; existing acceleration methods treat all state dimensions uniformly.
Approach
Observes that different hidden-state dimensions converge at very different speeds (heterogeneous convergence); stores past linear operator outputs and only propagates state changes exceeding a threshold, so converged dimensions are skipped in matrix-vector and convolution computations.
Key results
- On INR, DeltaDEQ-Fourier achieves about 84% FLOPs reduction with a minor PSNR drop (Fig. 3, 32.32 to 31.30 PSNR in the example).
- On optical flow with a 60-iteration schedule, DeltaDEQ with threshold 0.001 saves 77% (Sintel) and 66% (KITTI) of FLOPs with near-unchanged AEPE (1.24 vs 1.26 clean Sintel).
- With early stopping plus delta rule, average FLOPs fall below 20% of the original DEQ, and fine-tuned DeltaDEQ-ft (tau=0.005) improves Sintel final AEPE to 2.59 from 2.51 baseline range while cutting FLOPs.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Apply delta updates with threshold 0.001 | DEQ-RAFT-H · Sintel (train) | FLOPs per pair: 782G -> 264G | Helped |
| Apply delta updates with threshold 0.001 | DEQ-RAFT-H · KITTI (train) | FLOPs per pair: 814G -> 274G | Helped |
| Apply delta updates with threshold 0.001 | DEQ-RAFT-H · Sintel (train) | Clean EPE: 1.26 -> 1.24 | Helped |
| Apply delta updates with threshold 0.001 | DEQ-RAFT-H · KITTI (train) | AEPE: 3.73 -> 3.79 | Hurt |
| Fine-tune with delta threshold 0.005 (ablation) | DEQ-RAFT-H · Sintel (train) | Clean EPE: 1.48 -> 1.25 | Helped |
| Fine-tune with delta threshold 0.005 (ablation) | DEQ-RAFT-H · KITTI (train) | AEPE: 4.00 -> 3.86 | Helped |
| Use τ=0.005 instead of τ=0.001 (ablation) | DeltaDEQ · Sintel (train) | Clean EPE: 1.24 -> 1.48 | Hurt |
| Use τ=0.005 instead of τ=0.001 (ablation) | DeltaDEQ · KITTI (train) | AEPE: 3.79 -> 4.00 | Hurt |
| Apply the delta rule during training (ablation) | DEQ-Fourier · INR image reconstruction | Training FLOPs (Phantom Gradient): 140 -> 110 (-21%) | Helped |
| Apply the delta rule during training (ablation) | DEQ-Fourier · INR image reconstruction | PSNR (Phantom Gradient): 31.31 -> 31.06 | Hurt |
| Apply the delta rule during training (ablation) | DEQ-Siren · INR image reconstruction | Training FLOPs (IFT): 278 -> 190 (-32%) | Helped |
| Apply the delta rule during training (ablation) | DEQ-Siren · INR image reconstruction | PSNR (IFT): 34.96 -> 34.99 | No clear effect |
Surprising findings
- Even with threshold 0 (no thresholding), delta activation sparsity exceeds 50% in DEQ-RAFT due to ReLU activations.
- Fixed-point iterations (Picard, KM) outperformed Broyden and Anderson root-solvers in accuracy per iteration for DEQ-RAFT.
Limitations
- FLOPs savings are theoretical; translating them into wall-clock speedups requires specialized hardware or sparse kernels.
- Sparse convolution is harder to realize efficiently than sparse matrix-vector products.
- Evaluated only on INR and optical flow with small DEQ models; transformer applicability is discussed conceptually but not tested.
Experimental setup
Datasets: FlyingChairs FlyingThings3D MPI Sintel KITTI 2015 Kodak/Astronaut image (INR)
Benchmarks: MPI Sintel (train) KITTI 2015 (train) INR image reconstruction
Metrics: PSNR AEPE F1-all FLOPs reduction delta activation sparsity
Baselines: DEQ-Siren DEQ-Fourier DEQ(picard) RAFT GMA PWC-Net VCN
Builds on: deep equilibrium models DEQ-RAFT delta network update rule RAFT Siren Fourier filter networks
Closest papers by meaning
- On the Theory of Implicit Deep Learning: Global Convergence with Implicit Layers (ICLR 2021 · similarity 0.89)
- Positive Concave Deep Equilibrium Models (ICML 2024 · similarity 0.87)
- Separation and Bias of Deep Equilibrium Models on Expressivity and Learning Dynamics (NeurIPS 2024 · similarity 0.86)
- Subhomogeneous Deep Equilibrium Models (ICML 2024 · similarity 0.86)
- Deep Equilibrium Models are Almost Equivalent to Not-so-deep Explicit Models for High-dimensional Gaussian Mixtures (ICML 2024 · similarity 0.85)
- Implicit regularization of deep residual networks towards neural ODEs (ICLR 2024 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.