Research map / Learning theory and algorithms / Prediction and decision losses
Forward Learning with Top-Down Feedback: Empirical and Analytical Characterization
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Shows PEPITA approximates feedback alignment with an adaptive feedback matrix, analytically tracks its online learning dynamics in a teacher-student setup, extends it to deeper nets with weight mirroring, and relates a Hebbian variant of PEPITA to Forward-Forward.
Problem
Forward-only learning rules (PEPITA, Forward-Forward) lag behind backpropagation in accuracy, lack analytical understanding of their dynamics, and have biologically unrealistic aspects such as corrupted negative data and non-locality in time.
Approach
PEPITA's modulated update is approximated as feedback alignment with an adaptive feedback matrix (W1 F) that depends on forward weights (Adaptive Feedback Alignment); this enables ODE-based analysis of online teacher-student dynamics, a generalized weight-mirroring extension to PEPITA, and a Hebbian/anti-Hebbian rewrite linking PEPITA to Forward-Forward.
Key results
- PEPITA with weight mirroring and activation normalization reaches 53.80% on CIFAR-10 and 98.42% on MNIST with a 1x1024 network, versus 52.45% and 98.02% for vanilla PEPITA.
- Pre-mirroring plus WM reduces the PEPITA slowness on MNIST from 0.052 to 0.040, a faster convergence rate than BP (0.061) in that setting.
- Theoretical ODE predictions for the generalization error of AFA closely match PEPITA and AFA simulations in a teacher-student setup at D=500.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add weight decay to PEPITA (ablation) | PEPITA · CIFAR-10 | accuracy: 52.45$\pm$0.25 -> 53.05$\pm$0.23 | Helped |
| Add activation normalization to PEPITA (ablation) | PEPITA · CIFAR-10 | accuracy: 52.45$\pm$0.25 -> 53.51$\pm$0.23 | Helped |
| Add weight mirroring to PEPITA (ablation) | PEPITA · CIFAR-100 | accuracy: 24.69$\pm$0.17 -> 27.07$\pm$0.11 | Helped |
| Add weight mirroring and activation normalization to PEPITA (ablation) | PEPITA · CIFAR-10 | accuracy: 52.45$\pm$0.25 -> 53.80$\pm$0.25 | Helped |
| Use He normal instead of He uniform initialization (ablation) | PEPITA · CIFAR-10 | accuracy | No clear effect |
| Increase network depth without activation normalization (ablation) | PEPITA · CIFAR-10 | accuracy | No clear effect |
| Increase network depth with activation normalization (ablation) | PEPITA · CIFAR-10 | accuracy | Hurt |
| Replace PEPITA hidden-layer updates with the Hebbian approximation (ablation) | PEPITA · CIFAR-10 | accuracy | No clear effect |
| Make PEPITA updates time-local (ablation) | PEPITA · CIFAR-10 | accuracy | Hurt |
| Remove feedback by setting F=0 (ablation) | PEPITA-TL · CIFAR-10 | accuracy: approx. 40% -> approx. 18% | Hurt |
| Pre-mirror the feedback weights before training (ablation) | PEPITA + WM · MNIST | slowness: 0.047$\pm$0.005 -> 0.040$\pm$0.002 | Helped |
| Pre-mirror the feedback weights before training (ablation) | PEPITA + WM · CIFAR-10 | slowness: 0.890$\pm$0.081 -> 0.856$\pm$0.051 | Helped |
Surprising findings
- PEPITA's convergence on CIFAR-10 is slower than all baselines, including FA and DRTP, in the reported 1x1024 setting.
- The time-local PEPITA variant (PEPITA-TL) loses accuracy relative to standard PEPITA, reaching 92.78% on MNIST in the table.
Limitations
- Algorithms still lag behind backpropagation in accuracy and scalability.
- Weight mirroring factorization of F into layer-wise matrices may not correspond to a known biological mechanism.
- Theoretical analysis is limited to two-layer networks; deep-network dynamics are not analyzed.
Experimental setup
Datasets: MNIST CIFAR-10 CIFAR-100
Benchmarks: MNIST CIFAR-10 CIFAR-100
Metrics: test accuracy generalization error alignment angle slowness
Baselines: BP FA DRTP PEPITA Forward-Forward
Builds on: PEPITA Forward-Forward feedback alignment Weight Mirroring DRTP
Closest papers by meaning
- Overcoming Rank Collapse in Feedback Alignment (NeurIPS 2026 · similarity 0.89)
- Efficient Target Propagation by Deriving Analytical Solution (AAAI 2024 · similarity 0.88)
- Layer Collaboration in the Forward-Forward Algorithm (AAAI 2024 · similarity 0.88)
- Pretraining with Random Noise for Fast and Robust Learning without Weight Transport (NeurIPS 2024 · similarity 0.87)
- Forward Learning of Graph Neural Networks (ICLR 2024 · similarity 0.87)
- One-Time Soft Alignment Enables Resilient Learning without Weight Transport (NeurIPS 2026 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.