Research map / Architectures and efficiency / RNNs and normalization
MLAAN: Scaling Supervised Local Learning with Multilaminar Leap Augmented Auxiliary Network
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes MLAAN, a plug-and-play auxiliary-network design for supervised local learning that combines cascaded/independent auxiliary modules (MLM) with EMA-updated leap auxiliary modules (LAM) to close the gap with end-to-end backprop while reducing GPU memory.
Problem
Supervised local learning (gradient-isolated modules with auxiliary networks) lags behind end-to-end backpropagation in accuracy because local modules are myopic and lack global information and inter-module interaction.
Approach
Combine independent auxiliary networks (local features) with cascaded auxiliary networks shared across adjacent modules (global features), and add Leap Augmented Modules that deepcopy hidden layers from subsequent local modules into the auxiliary network and update them via EMA, so each module receives information from deeper layers.
Key results
- On CIFAR-10 with ResNet-32 (K=16), DGL+MLAAN reduces test error from 14.08 to 6.36, versus BP at 6.37.
- On ImageNet with ResNet-152 (K=51), MLAAN achieves 21.34 top-1 and 5.74 top-5 error versus BP at 21.60 and 5.92.
- On CIFAR-10 with ResNet-110 (K=55), GPU memory drops from 9.26G (BP) to 2.76G (-70.2%).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| integrate MLAAN with DGL | DGL · CIFAR-10, ResNet-32 K=16 | test error: 14.08 -> 6.36 | Helped |
| integrate MLAAN with DGL | DGL · STL-10, ResNet-32 K=16 | test error: 27.14 -> 19.96 | Helped |
| integrate MLAAN with DGL | DGL · SVHN, ResNet-32 K=16 | test error: 5.05 -> 2.82 | Helped |
| add MLAAN | BP, ResNet-34 · ImageNet validation | Top1 error: 25.38 -> 25.31(↓0.07) | No clear effect |
| add MLAAN | BP, ResNet-34 · ImageNet validation | Top5 error: 7.90 -> 7.88(↓0.02) | No clear effect |
| add MLAAN | BP, ImageNet ResNet-101 and ResNet-152 · ImageNet validation | Top1 error: 22.03 -> 21.69; 21.60 -> 21.34 | Helped |
| add MLAAN | BP, ImageNet ResNet-101 and ResNet-152 · ImageNet validation | Top5 error: 5.93 -> 5.72; 5.92 -> 5.74 | Mixed |
| add MLAAN | BP · CIFAR-10 and ImageNet | GPU memory: 3.37G -> 2.62G; 9.26G -> 2.76G; 10.74G -> 9.09G; 20.64G -> 17.93G; 26.29G -> 23.01G | Helped |
| add MLM without LAM (ablation) | DGL, ResNet-32 K=16 · CIFAR-10 | test error: 14.08 -> 8.46(↓5.62) | Helped |
| add LAM without MLM (ablation) | DGL, ResNet-32 K=16 · CIFAR-10 | test error: 14.08 -> 7.64(↓6.44) | Helped |
| add MLM and LAM together (ablation) | DGL, ResNet-32 K=16 · CIFAR-10 | test error: 14.08 -> 6.36(↓7.28) | Helped |
| add MLAAN and reduce training from 400 to 300 epochs (ablation) | DGL, ResNet-32 K=16 · CIFAR-10 | test error: 14.08 -> 7.79(↓6.29) | Helped |
Surprising findings
- MLAAN-augmented local learning surpasses end-to-end BP test error on CIFAR-10, STL-10, SVHN and ImageNet in the reported settings.
- Generalization from CIFAR-10 checkpoints to STL-10 is higher for DGL+MLAAN than for BP (e.g., 40.26 vs 36.78 for ResNet-110).
Limitations
- MLAAN adds extra wall-time cost per training run, mitigated only by an unequal-epoch comparison.
- Experiments are limited to image classification with ResNet backbones on four datasets.
- Hyperparameters such as the number of hidden layers p and cascade size K are chosen manually; results depend on K.
Experimental setup
Datasets: CIFAR-10 STL-10 SVHN ImageNet
Benchmarks: CIFAR-10 STL-10 SVHN ImageNet
Metrics: test error top-1 error top-5 error GPU memory CKA similarity linear separability
Baselines: BP DGL InfoPro
Builds on: DGL InfoPro CKA
Closest papers by meaning
- Scaling Supervised Local Learning with Augmented Auxiliary Networks (ICLR 2024 · similarity 0.87)
- Towards Interpretable Deep Local Learning with Successive Gradient Reconciliation (ICML 2024 · similarity 0.85)
- LAuReL: Learned Augmented Residual Layer (ICML 2025 · similarity 0.84)
- Local Loss Optimization in the Infinite Width: Stable Parameterization of Predictive Coding Networks and Target Propagation (ICLR 2025 · similarity 0.84)
- LEMON: Lossless model expansion (ICLR 2024 · similarity 0.83)
- A Multi-Level Framework for Accelerating Training Transformer Models (ICLR 2024 · similarity 0.83)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.