Research map / Architectures and efficiency / RNNs and normalization

Overcoming Rank Collapse in Feedback Alignment

NeurIPS 2026 · Architectures and efficiency › RNNs and normalization · approach: Backprop-free credit assignment · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Shows feedback alignment (FA) fails in deeper CNNs because its gradients and features collapse to low effective rank, and that Muon and batch normalisation raise dimensionality and substantially improve FA on CIFAR-10/100, STL-10 and Tiny ImageNet.

Problem

Feedback alignment, a biologically motivated alternative to backpropagation that avoids weight transport, performs poorly and does not scale to deeper CNNs and harder datasets; the paper asks what stops alignment from continuing.

Approach

FA gradients and trajectories have low effective rank, which blocks further weight alignment. Increasing update dimensionality with the Muon orthogonalised optimiser and increasing activity orthogonality with batch normalisation restores alignment and accuracy in deeper FA networks.

Key results

• On CIFAR-100 with ResNet-18, FA with Muon+BN reaches 46.1% vs 1.4% for baseline FA (BP baseline 71.9%).
• On STL-10 with AlexNet, FA with Muon+BN reaches 67.5% vs 34.2% baseline FA.
• On Tiny ImageNet with ResNet-18, FA Muon+BN reaches 24.3% vs 0.7% baseline FA.

Surprising findings

• Adding isotropic Gaussian update noise, which raises dimensionality, does not improve FA, suggesting extra dimensions must align with the learning signal.
• Low-rank (top-k) SGD-style updates severely harm FA while having little effect on BP.

Limitations

• A large gap to backpropagation remains on the hardest settings, e.g. 46.1% vs 71.9% on CIFAR-100 with ResNet-18.
• Muon and batch normalisation are not biologically plausible as implemented.
• Only two random seeds per configuration were used.

Experimental setup

Datasets: CIFAR-10 STL-10 CIFAR-100 Tiny ImageNet

Benchmarks: CIFAR-10 STL-10 CIFAR-100 Tiny ImageNet

Metrics: test accuracy effective rank gradient alignment weight alignment

Baselines: backpropagation feedback alignment baseline direct feedback alignment

Builds on: feedback alignment Muon batch normalization direct feedback alignment

Closest papers by meaning

• Forward Learning with Top-Down Feedback: Empirical and Analytical Characterization (ICLR 2024 · similarity 0.89)
• One-Time Soft Alignment Enables Resilient Learning without Weight Transport (NeurIPS 2026 · similarity 0.88)
• Implicit Regularization in Feedback Alignment Learning Mechanisms for Neural Networks (ICML 2024 · similarity 0.88)
• Efficient Target Propagation by Deriving Analytical Solution (AAAI 2024 · similarity 0.87)
• SDS-LoRA: Overcoming Anisotropic Gradient Scaling in Low-Rank Adaptation (NeurIPS 2026 · similarity 0.86)
• Efficient Backpropagation with Variance Controlled Adaptive Sampling (ICLR 2024 · similarity 0.86)