Research map / Architectures and efficiency / RNNs and normalization
A Stable, Fast, and Fully Automatic Learning Algorithm for Predictive Coding Networks
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes incremental predictive coding (iPC), which updates neural activities and weights in parallel at every step, giving faster, more stable PC training with convergence guarantees via incremental EM.
Problem
Predictive coding networks are slow and unstable to train: inference must converge before weight updates, requires an external control signal to switch phases, and convergence is sensitive to hyperparameters.
Approach
Update value nodes and synaptic weights simultaneously at every time step (a temporal rescheduling of the PC update derived from incremental EM), removing the inference-then-learning phase switch and the need for many inference iterations.
Key results
- On CIFAR-10 CNN, iPC reaches 72.54% vs 70.84% for PC and 69.34% for BP; on AlexNet CIFAR-10, PC reaches 64.63% vs iPC 72.42% and BP 75.64%.
- On masked LM, iPC gets 106.19 perplexity with all 10 seeds converging, versus 523.08 for PC (3 converged) and 120.02 for BP (7 converged).
- Theoretically, one full weight update of iPC takes O(1) sequential matrix-multiplication steps versus O(L) for BP in full-batch training.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Replace PC's sequential inference-then-learning schedule with simultaneous incremental updates | Predictive coding networks · FashionMNIST | test accuracy: 85.12%±0.75% -> 89.13%±0.86% | Helped |
| Replace PC's sequential inference-then-learning schedule with simultaneous incremental updates | Predictive coding networks · SVHN | test accuracy: 94.53%±1.54% -> 96.45%±1.04% | Helped |
| Replace PC's sequential inference-then-learning schedule with simultaneous incremental updates | Predictive coding networks · CIFAR-10, CNN | test accuracy: 70.84%±0.64% -> 72.54%±0.93% | Helped |
| Replace PC's sequential inference-then-learning schedule with simultaneous incremental updates | Predictive coding networks · CIFAR-10, AlexNet | test accuracy: 64.63%±1.55% -> 72.42%±0.53% | Helped |
| Replace PC's sequential inference-then-learning schedule with simultaneous incremental updates | Predictive coding networks · MNIST | test accuracy: 98.55%±0.14% -> 98.54%±0.86% | No clear effect |
| Replace PC's sequential inference-then-learning schedule with simultaneous incremental updates | Predictive coding networks · Masked LM | perplexity: 523.08 ± 12.3 -> 106.19 ± 10.54 | Helped |
| Replace PC's sequential inference-then-learning schedule with simultaneous incremental updates | Predictive coding networks · Masked LM | converged models: 3 -> 10 | Helped |
| Replace PC's sequential inference-then-learning schedule with simultaneous incremental updates | Predictive coding networks · Conditional LM | perplexity: 206.34 ± 6.46 -> 142.54 ± 4.23 | Helped |
| Use iPC instead of BP | BERT · Masked LM | perplexity: 120.02 ± 13.19 -> 106.19 ± 10.54 | Helped |
| Use iPC instead of BP | BERT · Conditional LM | perplexity: 113.32 ± 0.36 -> 142.54 ± 4.23 | Hurt |
| Increase CNN width using multiplier C (ablation) | iPC CNN · CIFAR-10 | final accuracy | Mixed |
| Use the F_KL modification for softmax layers instead of Gaussian distributions (ablation) | PC and iPC language models · Masked LM and Conditional LM | performance | Helped |
Surprising findings
- iPC on conditional LM reached 142.54 perplexity, worse than BP's 113.32, despite iPC outperforming BP on masked LM.
- In a CPU implementation, iPC is slower than BP on small architectures (hidden dimension <= 64) and only faster on large networks.
Limitations
- iPC's accuracy degrades at large width in the width experiment, requiring extra tuning of parameters.
- Wall-clock speedup over BP is not shown on GPUs; the naive CPU implementation has overhead on small networks.
- On conditional LM, iPC is worse than BP and has 3 non-converged seeds.
Experimental setup
Datasets: MNIST FashionMNIST SVHN CIFAR-10 Tiny ImageNet One Billion Word Benchmark
Benchmarks: MNIST FashionMNIST SVHN CIFAR-10 One Billion Word Benchmark
Metrics: test accuracy perplexity energy convergence AdaECE calibration error number of converged runs non-parallel matrix multiplications
Baselines: BP PC Z-IL AlexNet
Builds on: predictive coding incremental EM zero-divergence inference learning backpropagation
Closest papers by meaning
- Understanding and Improving Optimization in Predictive Coding Networks (AAAI 2024 · similarity 0.88)
- Benchmarking Predictive Coding Networks -- Made Simple (ICLR 2025 · similarity 0.88)
- Predictive Coding beyond Correlations (ICML 2024 · similarity 0.88)
- Divide-and-Conquer Predictive Coding: a structured Bayesian inference algorithm (NeurIPS 2024 · similarity 0.88)
- Training a Predictive Coding Network on ImageNet using Equilibrium Propagation (NeurIPS 2026 · similarity 0.87)
- Understanding Sample Efficiency in Predictive Coding (NeurIPS 2026 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.