Research map / Architectures and efficiency / RNNs and normalization
One-Time Soft Alignment Enables Resilient Learning without Weight Transport
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Shows that a one-time soft alignment of forward weights with fixed random feedback weights at initialization lets feedback alignment approach backpropagation performance without weight transport during training, with smoother loss landscapes and better corruption and adversarial robustness.
Problem
Feedback alignment avoids weight transport but performs poorly, especially in deep networks and with limited data, falling well short of backpropagation.
Approach
Initialize forward weights as a mixture of the fixed random feedback weights (rotated by an alignment angle) so forward and feedback weights start aligned, then let them evolve independently with no further synchronization; the initial angle controls trainability, flatness, and robustness.
Key results
- On CIFAR-10 with a multi-layer perceptron, IFA reaches final test accuracy not significantly different from BP (P=0.082, n=10) and significantly higher than FA (P<10^-3).
- Benchmarked accuracy (BP / FA 90° / IFA 0°): SVHN 82.90 / 78.78 / 81.91; CIFAR-10 53.32 / 47.21 / 53.04; CIFAR-100 24.18 / 16.79 / 23.31; STL-10 43.75 / 35.87 / 43.37.
- On CNNs, IFA (0°) improves over FA (90°) on CIFAR-10: 56.37% vs 45.67% for LeNet-variant and 65.55% vs 50.41% for AlexNet-variant, though still below BP (62.13% and 73.10%).
Surprising findings
- Moderate misalignment from exact symmetry improves FGSM, BIM and PGD adversarial robustness compared to standard backpropagation.
- Weaker initial alignment in FA produces higher loss-accuracy discrepancy in deep networks, suggesting calibration breakdown.
Limitations
- Performance gap to full backpropagation remains in deeper or more complex models, notably on CNNs.
- Evaluated mainly on small MLPs and standard image benchmarks.
- Biological plausibility of the initial alignment itself is acknowledged as not strictly established.
Experimental setup
Datasets: CIFAR-10 SVHN CIFAR-100 STL-10 CIFAR-10-C
Benchmarks: CIFAR-10 SVHN CIFAR-100 STL-10 CIFAR-10-C FGSM BIM PGD
Metrics: test accuracy training accuracy alignment angle Hessian trace Hessian top eigenvalue adversarial accuracy
Baselines: backpropagation feedback alignment sign-symmetry predictive coding equilibrium propagation Kolen-Pollack algorithm
Builds on: feedback alignment backpropagation sign-symmetry Kolen-Pollack algorithm
Closest papers by meaning
- Overcoming Rank Collapse in Feedback Alignment (NeurIPS 2026 · similarity 0.88)
- Pretraining with Random Noise for Fast and Robust Learning without Weight Transport (NeurIPS 2024 · similarity 0.87)
- Forward Learning with Top-Down Feedback: Empirical and Analytical Characterization (ICLR 2024 · similarity 0.87)
- Improving equilibrium propagation without weight symmetry through Jacobian homeostasis (ICLR 2024 · similarity 0.86)
- Sinusoidal Initialization, Time for a New Start (NeurIPS 2025 · similarity 0.85)
- Implicit Regularization in Feedback Alignment Learning Mechanisms for Neural Networks (ICML 2024 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.