Research map / Architectures and efficiency / RNNs and normalization
Towards Interpretable Deep Local Learning with Successive Gradient Reconciliation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes successive gradient reconciliation (SGR), a regularizer that aligns neighboring local layers' input-gradients to improve non-greedy local learning, matching global BP accuracy with 40%+ memory savings on ImageNet Transformers/CNNs.
Problem
Local (gradient-isolated) layer-wise training of neural networks underperforms global back-propagation because local errors of neighboring modules are not reconciled, so the last-layer loss may not converge; training also has high memory cost due to BP and update locking.
Approach
Add a regularizer per layer that minimizes the MSE between the input-gradient from the current local loss and the input-gradient from the previous local loss (passed forward as a detached tensor), reconciling neighboring local updates without breaking gradient isolation or adding learnable parameters; in BP-free mode, use fixed ETF classifiers.
Key results
- On ImageNet, SGR with Swin-small (K=3) reaches 81.64% top-1 vs 83.02% for global BP while cutting memory by 46.4% (42.60 to 22.83 GB).
- On CIFAR-10 with ResNet-32 and K=2, SGR reaches 92.91% vs 92.82% for global BP.
- In BP-free training on CIFAR-10 with PlainNet, SGR improves over layer-wise training from 69.17% to 72.40%.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add SGR to non-greedy local training (ablation) | ResNet-18 · ImageNet | top-1 accuracy: 69.44 -> 70.73 | Helped |
| Vary SGR strength from λ=1000 to λ=5000 (ablation) | ResNet-18 local training · ImageNet | top-1 accuracy: 70.32 -> 70.73 | Helped |
| Use SGR instead of layer-wise training (ablation) | ResNet-32 · CIFAR-10 | accuracy: 84.49$\pm$0.36 -> 85.65$\pm$0.38 | Helped |
| Use reforward instead of standard layer-wise training (ablation) | ResNet-32 · CIFAR-10 | accuracy: 84.49$\pm$0.36 -> 84.38$\pm$0.52 | No clear effect |
| Use SGR instead of layer-wise training (ablation) | PlainNet · CIFAR-10 | accuracy: 80.66$\pm$0.42 -> 81.67$\pm$0.29 | Helped |
| Use reforward instead of standard layer-wise training (ablation) | PlainNet · CIFAR-10 | accuracy: 80.66$\pm$0.42 -> 80.74$\pm$0.44 | No clear effect |
| Add SGR to BP-free layer-wise training (ablation) | PlainNet · CIFAR-10 | accuracy: 69.17$\pm$0.91 -> 72.40$\pm$0.75 | Helped |
| Add SGR to BP-free layer-wise training (ablation) | PlainNet · CIFAR-100 | accuracy: 48.30$\pm$0.64 -> 49.41$\pm$0.44 | Helped |
| Add SGR to ResNet-50 local training | ResNet-50, K=2 · ImageNet | memory: 21.49 -> 17.91 ($\downarrow$16.6\%) | Helped |
| Add SGR to Transformer local training | ViT-small · ImageNet | memory: 20.70 -> 11.73 ($\downarrow$43.3\%) | Helped |
| Add SGR to Transformer local training | Swin-small · ImageNet | top-1 accuracy: 83.02 -> 81.64 | Hurt |
Surprising findings
- Naively re-forwarding the updated output for each local update (reforward) gives no gain and slightly hurts ResNet results, attributed to overfitting from extra optimization steps.
- Local learning with SGR stays stable on a 57-layer VGG where global BP fails to converge (37.68% at depth 47, fail at 57).
Limitations
- Training time can be substantially higher than global BP for some configurations (e.g., ResNet-50 K=2: 17.1 h vs 7.0 h)
- Theoretical analysis relies on a two-layer model with PL condition and Lipschitz assumptions, and the BP-free proposition is for linear models
- Hyperparameter lambda must be tuned, and the gap to global BP remains for some architectures (e.g., ResNet-50 K=3 75.42 vs 76.55)
Experimental setup
Datasets: ImageNet CIFAR-10 CIFAR-100
Benchmarks: ImageNet CIFAR-10 CIFAR-100
Metrics: top-1 accuracy top-5 accuracy training memory (GB) training time (hours)
Baselines: Global BP Greedy layer-wise DGL InfoPro FA DRTP PEPITA
Builds on: InfoPro DGL DRTP PEPITA feedback alignment
Closest papers by meaning
- Local Loss Optimization in the Infinite Width: Stable Parameterization of Predictive Coding Networks and Target Propagation (ICLR 2025 · similarity 0.88)
- BP-Modified Local Loss for Efficient Training of Deep Neural Networks (ICLR 2025 · similarity 0.87)
- Scaling Supervised Local Learning with Augmented Auxiliary Networks (ICLR 2024 · similarity 0.85)
- MLAAN: Scaling Supervised Local Learning with Multilaminar Leap Augmented Auxiliary Network (AAAI 2025 · similarity 0.85)
- The Price of Locality: Why Forward-Forward Underperforms Backpropagation? (NeurIPS 2026 · similarity 0.85)
- Role of Locality and Weight Sharing in Image-Based Tasks: A Sample Complexity Separation between CNNs, LCNs, and FCNs (ICLR 2024 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.