Research map / Learning theory and algorithms / Kernels and regression theory
Understanding Sample Efficiency in Predictive Coding
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Analytically shows PC's weight updates in deep linear networks yield higher target alignment (less interference) than BP, and proposes rescaled PC variants that achieve optimal alignment, improving sample efficiency.
Problem
Why and when predictive coding (PC) is more sample efficient than backpropagation (BP) is not analytically understood; existing evidence is small-scale.
Approach
Defines target alignment as cosine similarity between output error and change in prediction; derives closed-form prediction changes for BP and PC in deep linear networks, showing PC's inverse scaling matrix S^-1 cancels weight-dependent interference, and proposes layer-wise adaptive learning rates and batch decorrelation rescaling that give optimal alignment.
Key results
- PC achieves higher target alignment than BP after one update in deep, narrow DLNs, with alignment for BP degrading as weight condition number increases while PC with adaptive learning rates stays at alignment 1.
- Weight-update rescaled PC achieves optimal target alignment independent of batch size, while BP with scaling still underperforms PC.
- With norm-preserving initialisation, PC reaches lower weight error than BP over whole training trajectories for linear networks, including 8-hidden-layer networks.
Surprising findings
- Scaling learning rates gives little sample-efficiency benefit early in online (batch size 1) training, because the single-sample Fisher estimate loses its geometric advantage.
- Kaiming uniform initialisation (PyTorch default) causes activity norm decay with depth, placing PC outside its ideal regime.
Limitations
- Theory assumes linear networks and equilibrated PC energy; nonlinear results are preliminary.
- Rescaling methods are biologically implausible and serve as optimal performance baselines rather than practical methods.
- Analysis focuses on MLPs; residual and convolutional networks are not studied extensively.
Experimental setup
Datasets: MNIST synthetic linear regression data
Benchmarks: MNIST
Metrics: target alignment (cosine similarity) weight error ||W_data - W_L:1||^2 test reconstruction error
Baselines: backpropagation predictive coding scaled BP scaled PC
Builds on: predictive coding backpropagation natural gradient ePC target propagation
Closest papers by meaning
- Benchmarking Predictive Coding Networks -- Made Simple (ICLR 2025 · similarity 0.89)
- $\mu$PC: Scaling Predictive Coding to 100+ Layer Networks (NeurIPS 2025 · similarity 0.88)
- Divide-and-Conquer Predictive Coding: a structured Bayesian inference algorithm (NeurIPS 2024 · similarity 0.88)
- Understanding and Improving Optimization in Predictive Coding Networks (AAAI 2024 · similarity 0.88)
- Only Strict Saddles in the Energy Landscape of Predictive Coding Networks? (NeurIPS 2024 · similarity 0.88)
- A Stable, Fast, and Fully Automatic Learning Algorithm for Predictive Coding Networks (ICLR 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.