Research map / Architectures and efficiency / RNNs and normalization

The Price of Locality: Why Forward-Forward Underperforms Backpropagation?

NeurIPS 2026 · Architectures and efficiency › RNNs and normalization · approach: Layer-wise analysis of language models · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Theoretically analyzes why Forward-Forward (FFA) underperforms backpropagation, proving local PL convergence to an error plateau from concurrent layer updates and kernel contraction toward rank one that bounds FFA's effective learning capacity independent of depth.

Problem

Persistent and depth-growing performance gap between Forward-Forward local learning and backpropagation; the mechanism behind FFA's failure at scale is not understood.

Approach

Decomposes FFA's shortfall into an optimization floor from concurrent local updates (local PL convergence to a coupling error plateau) and a representational collapse where the layer Gram kernel contracts toward rank one, bounding FFA's effective learning capacity while BP's chain-rule error signal preserves diversity that scales with depth.

Key results

• On CIFAR-10 with ResNet-18/24/56, BP achieves 0.95-0.96 accuracy while local methods (LCE, SFF, DF) trail by roughly 5-8 pp and goodness-based FFA variants reach only 0.31-0.42.
• In Transformer pre-training on OpenWebText, the FFA/BP perplexity ratio widens from 3.73x (7M params) to 9.19x (124M params), while LCE/BP goes from 1.20x to 1.51x.
• In a controlled 12-layer CNN partition study, the strictly local 12x1 partition reaches 53.02% best accuracy versus 78.74% for the fully global 1x12 partition and 85.67% for BP.

Surprising findings

• Interpolating the BP/FFA update spectrum to a flat, Muon-like spectrum improves FFA by about 10 points but still leaves a 21.23-point gap to BP.

Limitations

• Theoretical guarantees are local (within a neighborhood of initialization) and rely on lazy/NTK-style assumptions that may not hold in feature-learning regimes.
• Kernel contraction result relies on i.i.d. kernel-product hypotheses and a BP Jacobian concentration assumption that is strong and only empirically supported.
• The error plateau is only an upper bound; whether it is an analysis artifact is left open.

Experimental setup

Datasets: CIFAR-10 OpenWebText

Benchmarks: CIFAR-10 OpenWebText

Metrics: test accuracy perplexity final loss effective rank Lyapunov sub-optimality

Baselines: Forward-Forward SymBa SCFF Trifecta Layer Collaboration Local Cross-Entropy Scalable Forward-Forward Distance-Forward backpropagation

Builds on: Forward-Forward algorithm neural tangent kernel Polyak-Lojasiewicz inequality

Closest papers by meaning

• Forward Learning with Top-Down Feedback: Empirical and Analytical Characterization (ICLR 2024 · similarity 0.87)
• Layer Collaboration in the Forward-Forward Algorithm (AAAI 2024 · similarity 0.86)
• Reducing Fine-Tuning Memory Overhead by Approximate and Memory-Sharing Backpropagation (ICML 2024 · similarity 0.85)
• Stochastic Forward-Forward Learning through Representational Dimensionality Compression (NeurIPS 2025 · similarity 0.85)
• Convolutional Channel-Wise Competitive Learning for the Forward-Forward Algorithm (AAAI 2024 · similarity 0.85)
• Overcoming Rank Collapse in Feedback Alignment (NeurIPS 2026 · similarity 0.85)