Research map / Architectures and efficiency / RNNs and normalization
Layer Collaboration in the Forward-Forward Algorithm
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Collaborative Forward-Forward, which adds the global goodness of other layers to each layer's forward-forward objective (and alternates layer updates) so layers can collaborate, and links the method to functional entropy.
Problem
The forward-forward algorithm trains each layer independently, so layers do not collaborate: early layers cannot access information from later layers, and the first layer alone often performs best, limiting hierarchical representation learning.
Approach
Add a constant term equal to the summed goodness of the other layers (computed with frozen copies) to each layer's positive/negative sigmoid objective, and alternate updates across all layers, so each layer's gradient is informed by the global network goodness while keeping forward-forward's no-backprop assumptions.
Key results
- On CIFAR-10 the collaborative forward-forward (network-level training, gamma<t) reaches 51.6% error versus 53.7% for classical forward-forward network training, and 53.2% vs 54.2% for layer-loss variants.
- On Fashion-MNIST, collaborative forward-forward with gamma reduces error from 13.6% (classical FF) to 12.0%.
- Backpropagation baselines remain best, e.g., 1.5% on MNIST and 45.4-45.9% on CIFAR-10, leaving a gap to backpropagation-level performance.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add collaboration using γ<t and alternate updates across layers | Forward-Forward · MNIST | error (%): 3.4 -> 2.2 | Helped |
| Add collaboration using γ<t and alternate updates across layers | Forward-Forward · Fashion-MNIST | error (%): 13.6 -> 11.6 | Helped |
| Add collaboration using γ<t and alternate updates across layers | Forward-Forward · CIFAR-10 | error (%): 53.7 -> 51.6 | Helped |
| Use γ<t instead of γ from all other layers (ablation) | Collaborative Forward-Forward, Network · MNIST | error (%): 2.1 -> 2.2 | Hurt |
| Use γ<t instead of γ from all other layers (ablation) | Collaborative Forward-Forward, Network · Fashion-MNIST | error (%): 12.0 -> 11.6 | Helped |
| Use γ<t instead of γ from all other layers (ablation) | Collaborative Forward-Forward, Network · CIFAR-10 | error (%): 52.7 -> 51.6 | Hurt |
| Optimize functional entropy instead of the sigmoid objective (ablation) | Forward-Forward, Layer · Fashion-MNIST | error (%): 13.2 -> 13.7% | Hurt |
| Optimize functional entropy with the collaborative loss (ablation) | Collaborative Forward-Forward · Fashion-MNIST | error (%): 12.0 -> 12.9% | Hurt |
Surprising findings
- In classical forward-forward, the first layer alone achieves the lowest error and adding later layers increases error, indicating poor layer collaboration.
- Replacing the sigmoid and threshold with a functional-entropy objective yields results on par with the classical and collaborative forward-forward.
Limitations
- Forward-forward remains far from backpropagation performance on these benchmarks.
- Not yet demonstrated on complex applications such as larger image classification, text, generative tasks, or modern architectures (convnets, transformers).
Experimental setup
Datasets: MNIST Fashion-MNIST CIFAR-10
Benchmarks: MNIST Fashion-MNIST CIFAR-10
Metrics: classification error functional entropy
Baselines: Forward-Forward (layer loss) Backpropagation Pairwise Backpropagation Classic
Builds on: Forward-Forward algorithm backpropagation energy-based models
Closest papers by meaning
- Forward Learning with Top-Down Feedback: Empirical and Analytical Characterization (ICLR 2024 · similarity 0.88)
- The Price of Locality: Why Forward-Forward Underperforms Backpropagation? (NeurIPS 2026 · similarity 0.86)
- Forward Learning of Graph Neural Networks (ICLR 2024 · similarity 0.86)
- Feature Learning Dynamics in Infinite-Depth Neural Networks (NeurIPS 2026 · similarity 0.84)
- Fast and Slow Gradient Approximation for Binary Neural Network Optimization (AAAI 2025 · similarity 0.84)
- On The Concurrence of Layer-wise Preconditioning Methods and Provable Feature Learning (ICML 2025 · similarity 0.83)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.