Research map / Architectures and efficiency / RNNs and normalization
Enforcing Idempotency in Neural Networks
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes "Modified Backpropagation," which replaces the gradient of an idempotency loss with the perturbation-theory corrector 3K^2-2K^3 (applied to network outputs), enabling architecture-agnostic training of idempotent networks, and replicates idempotent generative networks on MNIST and CelebA.
Problem
Training neural networks f: R^n -> R^n to be idempotent (f(x)=f(f(x))) with standard gradient descent on an idempotency MSE loss gives relatively poor reduction in idempotent error and is computationally costly without memoization.
Approach
Derives an idempotent corrector g(K)=3K^2-2K^3 via perturbation theory and uses its output residual 3f(y)-2f(f(y))-y as a substitute for the analytic gradient of the idempotent loss in backpropagation, giving an architecture-agnostic training scheme; for linear single-layer nets the recurrence K'=K+γ(g(K)-K) is shown to have idempotent attracting fixed points.
Key results
- On MLP networks B2-B4, Modified Backpropagation achieves significantly lower idempotent error than Ordinary Backpropagation at lower learning rates; for B3 the difference exceeds one order of magnitude.
- Modified Backpropagation has roughly the same wall-clock time per epoch as Ordinary Backpropagation for B1-B4 (Figure 14).
- The U-net DCGAN trained with Modified Backpropagation reproduces qualitative MNIST and CelebA results of Shocher et al. 2023, though CelebA results are less competitive.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use Modified Backpropagation instead of Ordinary Backpropagation | MLP B2 · B2 | absolute idempotent error | Helped |
| Use Modified Backpropagation instead of Ordinary Backpropagation | MLP B3 · B3 | absolute idempotent error: more than one order of magnitude | Helped |
| Use Modified Backpropagation instead of Ordinary Backpropagation | MLP B4 · B4 | absolute idempotent error | Helped |
Surprising findings
- Modified Backpropagation follows gradient directions substantially dissimilar from those of Ordinary Backpropagation, yet still reduces idempotent error more effectively.
- Non-idempotent fixed points with eigenvalue 0.5 exist but are repelling under the recurrence, so the corrector is attracting only around idempotent points.
Limitations
- Theoretical justification only holds in the linearized setting; non-linear case is heuristic.
- Generative results on CelebA are less competitive and the method is highly sensitive to loss weighting hyperparameters.
- Experiments are limited to MLP and one DCGAN architecture on synthetic and small image datasets.
Experimental setup
Datasets: MNIST CelebA
Benchmarks: MNIST CelebA
Metrics: idempotent error (MSE) reconstruction loss
Baselines: Ordinary Backpropagation
Builds on: Idempotent Generative Network (Shocher et al. 2023) backpropagation Kato perturbation theory
Closest papers by meaning
- Idempotent Generative Network (ICLR 2024 · similarity 0.89)
- Identifiability of Deep Polynomial Neural Networks (NeurIPS 2025 · similarity 0.84)
- Idempotence and Perceptual Image Compression (ICLR 2024 · similarity 0.82)
- PowerMLP: An Efficient Version of KAN (AAAI 2025 · similarity 0.81)
- MgNO: Efficient Parameterization of Linear Operators via Multigrid (ICLR 2024 · similarity 0.80)
- Beyond ICA: Identifiability by Symmetry Breaking (NeurIPS 2026 · similarity 0.80)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.