Research map / Architectures and efficiency / RNNs and normalization
Scaling Supervised Local Learning with Augmented Auxiliary Networks
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes AugLocal, a supervised local learning method whose per-layer auxiliary networks are built from uniformly sampled subsequent primary-network layers with pyramidal depth, closing much of the gap to backpropagation on CIFAR-10, SVHN, STL-10 and ImageNet.
Problem
Supervised local learning, which trains each layer with a gradient-isolated auxiliary network, lags far behind backpropagation in accuracy on large-scale deep networks because local layers lack feedback from subsequent layers.
Approach
Build each hidden layer's auxiliary network from a uniformly selected small subset of its subsequent primary-network layers (sharing their architecture, not weights) to enhance synergy, and linearly shrink auxiliary depth for deeper layers to cut computation.
Key results
- On CIFAR-10 with ResNet-110 (55 local layers), AugLocal (d=6) reaches 93.96% vs BP 94.61% and DGL 85.69%.
- AugLocal reduces GPU memory by up to 81.5% on ResNet-110 CIFAR-10 and 32-38% on ImageNet ResNets relative to BP (the 40% headline figure is an average claim in the abstract).
- On ImageNet, AugLocal with VGG13 reaches 70.92% top-1 vs BP 71.59% and DGL 67.32%.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use AugLocal with augmented auxiliary networks instead of other supervised local learning rules | ResNet-32 and ResNet-110 · CIFAR-10, SVHN, and STL-10 | test accuracy | Helped |
| increase auxiliary-network depth from d=2 to d=6 (ablation) | AugLocal on ResNet-110 · CIFAR-10 | test accuracy: 90.98$\pm$0.05 -> 93.96$\pm$0.15 | Helped |
| increase auxiliary-network depth from d=2 to d=6 (ablation) | AugLocal on ResNet-32 · CIFAR-10 | test accuracy: 91.12$\pm$0.24 -> 93.47$\pm$0.09 | Helped |
| use uniform layer selection rather than sequential selection (ablation) | AugLocal auxiliary networks · CIFAR-10, ResNet-110, d=2 | test accuracy: 82.19$\pm$0.47 -> 90.98$\pm$0.05 | Helped |
| use uniform layer selection rather than repetitive selection (ablation) | AugLocal auxiliary networks · CIFAR-10, ResNet-110, d=2 | test accuracy: 82.24$\pm$0.25 -> 90.98$\pm$0.05 | Helped |
| use augmented auxiliary networks rather than 1×1 or 3×3 convolution auxiliary networks (ablation) | AugLocal on ResNet-110 · CIFAR-10, d=6 | test accuracy: 88.82$\pm$0.09; 90.01$\pm$0.05 -> 93.96$\pm$0.15 | Helped |
| add downsampling when replacing Stage 2 uniform selection with repetitive selection (ablation) | AugLocal on ResNet-110 · CIFAR-10, d=6 | test accuracy: 91.73$\pm$0.11 -> 92.75$\pm$0.09 | Helped |
| retain uniform selection in Stage 2 rather than repetitive selection with downsampling (ablation) | AugLocal on ResNet-110 · CIFAR-10, d=6 | test accuracy: 92.75$\pm$0.09 -> 93.96$\pm$0.15 | Helped |
| use pyramidal auxiliary-network depth (ablation) | AugLocal on ResNet-110 · CIFAR-10 | auxiliary-network FLOPs: 45% | Helped |
| increase the pyramidal-depth decay factor (ablation) | AugLocal on ResNet-110 · CIFAR-10 | test accuracy: around 0.5% | Hurt |
| add the InfoPro reconstruction loss (ablation) | AugLocal on ResNet-110 · CIFAR-10 | test accuracy: 90.98$\pm$0.05 -> 91.59$\pm$0.11; 93.96$\pm$0.15 -> 94.02$\pm$0.09; 94.30$\pm$0.17 -> 94.08$\pm$0.18 | Mixed |
| train with AugLocal rather than end-to-end BP | VGG13, ResNet-34, and ResNet-101 · ImageNet validation | Top-1 Acc.: 71.59 -> 70.92; 74.28 -> 73.95; 77.34 -> 76.70 | Hurt |
Surprising findings
- A single hidden layer plus linear classifier (d=2) in AugLocal already outperforms deeper handcrafted auxiliary networks of DGL and InfoPro on CIFAR-10 ResNet-110.
Limitations
- Wall-clock training is slower than BP without a dedicated parallel implementation; theoretical speedup not realized.
- Auxiliary networks still introduce update locking as their depth grows.
- Evaluated mainly on image classification with CNNs; no transformers or language tasks.
Experimental setup
Datasets: CIFAR-10 SVHN STL-10 ImageNet COCO
Benchmarks: CIFAR-10 SVHN STL-10 ImageNet COCO
Metrics: top-1 accuracy top-5 accuracy GPU memory FLOPs CKA similarity linear probing accuracy AP
Baselines: BP PredSim InfoPro DGL gradient checkpointing
Builds on: DGL PredSim InfoPro stochastic depth
Closest papers by meaning
- MLAAN: Scaling Supervised Local Learning with Multilaminar Leap Augmented Auxiliary Network (AAAI 2025 · similarity 0.87)
- Towards Interpretable Deep Local Learning with Successive Gradient Reconciliation (ICML 2024 · similarity 0.85)
- Local Loss Optimization in the Infinite Width: Stable Parameterization of Predictive Coding Networks and Target Propagation (ICLR 2025 · similarity 0.84)
- $\mu$PC: Scaling Predictive Coding to 100+ Layer Networks (NeurIPS 2025 · similarity 0.82)
- Role of Locality and Weight Sharing in Image-Based Tasks: A Sample Complexity Separation between CNNs, LCNs, and FCNs (ICLR 2024 · similarity 0.82)
- Augmented Lagrangian Predictive Coding (NeurIPS 2026 · similarity 0.82)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.