Research map / Learning theory and algorithms / Prediction and decision losses
Error Broadcast and Decorrelation as a Potential Artificial and Natural Learning Mechanism
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Error Broadcast and Decorrelation (EBD), which trains layers by decorrelating broadcast output errors from layer activations, grounded in MMSE orthogonality, and applies it to MLP/CNN/LC networks and CorInfoMax biological networks.
Problem
Credit assignment in neural networks without weight transport: backpropagation requires symmetric backward weights, which is biologically implausible, while existing error-broadcast methods like DFA use fixed random projections.
Approach
Layer-wise losses penalize the cross-correlation between layer activations (or nonlinear functions of them) and the output error, motivated by the stochastic orthogonality of MMSE estimator errors; the projection weights are learned via a running cross-correlation estimate, yielding local three-factor updates with broadcast error.
Key results
- EBD matches DFA-family methods on MNIST (MLP 98.2%, CNN 99.1%, LC 98.9%) but BP remains best there.
- On CIFAR-100 CNN, EBD reaches 45.9% vs. 41.9% for DFA, while BP with cross-entropy reaches 60.5%.
- 10-layer CorInfoMax-EBD reaches 96.38% on MNIST and 54.89% on CIFAR-10 with online learning (batch size 1).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use EBD instead of DFA | MNIST MLP · MNIST | accuracy (%): 98.1{\pm0.21} -> 98.2{\pm0.08} | No clear effect |
| Use EBD instead of DFA | MNIST CNN · MNIST | accuracy (%): 99.1{\pm0.05} -> 99.1{\pm0.07} | No clear effect |
| Use EBD instead of DFA | MNIST LC · MNIST | accuracy (%): 98.9{\pm0.03} -> 98.9{\pm0.04} | No clear effect |
| Use EBD instead of the strongest error-broadcast baseline, NN-GEVB | CIFAR-10 MLP · CIFAR-10 | accuracy (%): 52.4 -> 55.5 | Mixed |
| Use EBD instead of the strongest error-broadcast baseline, NN-GEVB | CIFAR-10 CNN · CIFAR-10 | accuracy (%): 66.3 -> 66.4{\pm0.43} | No clear effect |
| Use EBD instead of the strongest error-broadcast baseline, DFA | CIFAR-10 LC · CIFAR-10 | accuracy (%): 62.2{\pm0.21} -> 64.3{\pm0.26} | Helped |
| Use EBD instead of BP | CIFAR-100 CNN · CIFAR-100 | accuracy (%): 60.5{\pm0.17} -> 45.9{\pm0.69} | Hurt |
| Replace CorInfoMax-EP with CorInfoMax-EBD | CorInfoMax, 3 layers, batch size 20 · CIFAR-10 | accuracy (%): 51.0 -> 55.7{\pm0.17} | Helped |
| Replace CorInfoMax-EP with CorInfoMax-EBD | CorInfoMax, 3 layers, batch size 20 · MNIST | accuracy (%): 97.6 -> 97.5{\pm0.12} | No clear effect |
| Use 10 layers instead of 3 layers | CorInfoMax-EBD, batch size 1 · CIFAR-10 | accuracy (%): 53.4{\pm0.33} -> 54.9{\pm0.58} | Helped |
| Use 10 layers instead of 3 layers | CorInfoMax-EBD, batch size 1 · MNIST | accuracy (%): 94.9{\pm0.16} -> 96.4{\pm0.11} | Helped |
Surprising findings
- Decorrelation between output error and activations also appears in backpropagation-trained networks, suggesting decorrelation may be a general feature of learning.
Limitations
- Evaluated only on MNIST and CIFAR-10/100 image classification; scalability to larger datasets untested.
- Additional computational and memory overhead from dynamic projection matrices and regularizers versus DFA.
- Several hyperparameters (learning rates, forgetting factors) require tuning; BP still outperforms EBD on CIFAR-10/100.
Experimental setup
Datasets: MNIST CIFAR-10 CIFAR-100
Benchmarks: MNIST CIFAR-10 CIFAR-100
Metrics: test accuracy
Baselines: backpropagation DFA DFA+E NN-GEVB MS-GEVB CorInfoMax-EP BP (cross-entropy)
Builds on: direct feedback alignment CorInfoMax global error vector broadcasting entropy regularization of layer activations
Closest papers by meaning
- Credit Assignment Through Broadcasting a Global Error Vector (NeurIPS 2021 · similarity 0.87)
- Error Feedback Reloaded: From Quadratic to Arithmetic Mean of Smoothness Constants (ICLR 2024 · similarity 0.84)
- Improving equilibrium propagation without weight symmetry through Jacobian homeostasis (ICLR 2024 · similarity 0.83)
- Counter-Current Learning: A Biologically Plausible Dual Network Approach for Deep Learning (NeurIPS 2024 · similarity 0.82)
- Outliers with Opposing Signals Have an Outsized Effect on Neural Network Optimization (ICLR 2024 · similarity 0.82)
- Forward Learning with Top-Down Feedback: Empirical and Analytical Characterization (ICLR 2024 · similarity 0.82)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.