Research map / Architectures and efficiency / RNNs and normalization
Implicit Regularization in Feedback Alignment Learning Mechanisms for Neural Networks
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proves a conservation law for (sign) feedback alignment in ReLU networks that implies implicit regularization keeping forward and feedback weights aligned, defines alignment dominance to give convergence guarantees, and shows better alignment (adaFA, sign-FA) helps multi-class performance on CIFAR-100 and Tiny-ImageNet subsets.
Problem
Feedback Alignment methods avoid weight transport and are biologically plausible, but they underperform on multi-class classification and their alignment mechanism and convergence are poorly understood theoretically.
Approach
A neuron-level conservation law shows that under (sign) FA the change in forward-feedback inner product for each neuron equals the change in half the squared norm of incoming weights, giving implicit regularization toward alignment; an alignment-dominance condition (analogous to the PL inequality) then yields convergence for two-layer nets.
Key results
- On CIFAR-100 with 50 classes, signFA reaches 48.9% ± 0.3% test accuracy versus 50.6% for GD and 32.6% for FA.
- On Tiny-ImageNet with 50 classes, signFA achieves 47.7% ± 0.6%, versus 53.4% for GD and 30.9% for FA.
- Measured deviation from the conservation law shrinks with width; for width 200 the mean absolute deviation is 0.01 for FA, 0.004 for adaFA and 0.003 for uSF, while GD deviates by 0.2.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use signFA instead of FA | LeNet · CIFAR-100 10-class subset | test accuracy: 55.2% ± 3.8% -> 74.1% ± 0.1% | Helped |
| Use signFA instead of FA | LeNet · CIFAR-100 25-class subset | test accuracy: 39.7 % ± 0.5% -> 58.9% ± 1.2% | Helped |
| Use signFA instead of FA | LeNet · CIFAR-100 50-class subset | test accuracy: 32.6 % ± 0.6% -> 48.9 % ± 0.3% | Helped |
| Use signFA instead of adaFA initialization | LeNet · CIFAR-100 10-class subset | test accuracy: 61.6% ± 0.4% -> 74.1% ± 0.1% | Helped |
| Use signFA instead of adaFA initialization | LeNet · CIFAR-100 25-class subset | test accuracy: 42.9% ± 1.2% -> 58.9% ± 1.2% | Helped |
| Use signFA instead of adaFA initialization | LeNet · CIFAR-100 50-class subset | test accuracy: 34.1 % ± 0.5% -> 48.9 % ± 0.3% | Helped |
| Use signFA instead of FA | ResNet-18 · Tiny-ImageNet 10-class subset | test accuracy: 49.6% ± 1.7% -> 64.9% ± 4.8% | Helped |
| Use signFA instead of FA | ResNet-18 · Tiny-ImageNet 25-class subset | test accuracy: 40.1 % ± 1.6% -> 53.0% ± 2.6% | Helped |
| Use signFA instead of FA | ResNet-18 · Tiny-ImageNet 50-class subset | test accuracy: 30.9 % ± 0.8% -> 47.7 % ± 0.6% | Helped |
| Use signFA instead of adaFA initialization | ResNet-18 · Tiny-ImageNet 10-class subset | test accuracy: 51.6% ± 2.2% -> 64.9% ± 4.8% | Helped |
| Use signFA instead of adaFA initialization | ResNet-18 · Tiny-ImageNet 25-class subset | test accuracy: 35.3% ± 1.0% -> 53.0% ± 2.6% | Helped |
| Use signFA instead of adaFA initialization | ResNet-18 · Tiny-ImageNet 50-class subset | test accuracy: 31.0 % ± 0.9% -> 47.7 % ± 0.6% | Helped |
Surprising findings
- Sign-FA stays competitive with backpropagation on multi-class tasks, while adaFA's initialization advantage diminishes as class count grows.
- Alignment in sign-FA shows non-monotonic behavior, and alignment degrades as the number of classes grows.
Limitations
- Convergence guarantees are proved only for two-layer leaky ReLU networks with exponential loss on orthogonally separable or nearly orthogonal data.
- Theory bounds alignment but does not explain why alignment increases over time.
- Deep-network convergence and the kernel-regime alignment-dominance case are not developed.
Experimental setup
Datasets: MNIST CIFAR-100 Tiny-ImageNet
Benchmarks: CIFAR-100 n-class subsets Tiny-ImageNet n-class subsets noisy MNIST
Metrics: test accuracy test loss alignment cosine conservation-law deviation
Baselines: FA adaFA signFA GD
Builds on: feedback alignment sign-concordant feedback alignment direct feedback alignment Polyak-Lojasiewicz inequality
Closest papers by meaning
- Overcoming Rank Collapse in Feedback Alignment (NeurIPS 2026 · similarity 0.88)
- Forward Learning with Top-Down Feedback: Empirical and Analytical Characterization (ICLR 2024 · similarity 0.87)
- One-Time Soft Alignment Enables Resilient Learning without Weight Transport (NeurIPS 2026 · similarity 0.85)
- Pretraining with Random Noise for Fast and Robust Learning without Weight Transport (NeurIPS 2024 · similarity 0.85)
- Learning Input Encodings for Kernel-Optimal Implicit Neural Representations (ICML 2025 · similarity 0.85)
- Gradient-Weight Alignment as a Train-Time Proxy for Generalization in Classification Tasks (NeurIPS 2025 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.