Research map / Architectures and efficiency / RNNs and normalization
Understanding and Improving Optimization in Predictive Coding Networks
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes sequential inference to cheapen predictive coding inference learning, a custom Matrix Update Equalization (MQ) optimizer for IL, and theory linking IL to implicit SGD and second-order information.
Problem
Inference learning (IL) in predictive coding networks is computationally expensive, prone to poor local minima without memory-heavy optimizers like Adam, and its speedups over backpropagation are not well understood.
Approach
Update activities sequentially from output to input so errors propagate to all layers in one iteration (allowing small T), pair with a memory-light per-layer scalar-learning-rate optimizer (MQ) that equalizes update magnitudes, and show theoretically that IL approximates implicit SGD and is sensitive to second-order information.
Key results
- SeqIL-MQ reaches 84.51% on SVHN FC and 57.70% on CIFAR-10 FC, matching or slightly exceeding BP (84.40%, 56.55%) and comparable to BP-Adam.
- On Tiny ImageNet, SeqIL-MQ achieves 23.91% top-1 vs 23.14% for BP and 25.61% for BP-Adam.
- SeqIL-MQ reduces loss faster than BP-SGD early in training in all simulations while using about one third the memory of Adam per parameter.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add the MQ optimizer to sequential inference (ablation) | SeqIL · SVHN, CIFAR-10, and Tiny ImageNet classification | test accuracy | Helped |
| remove the optimizer (ablation) | SeqIL-MQ · SVHN, CIFAR-10, and Tiny ImageNet classification | test accuracy: 84.51(±.17) -> 82.19(±.70); 88.67(±.18) -> 81.55(±.59); 57.70(±.21) -> 54.24(±.18); 67.50(±.19) -> 50.67(±.76); 23.91(±. | Hurt |
| replace MQ with Adam (ablation) | SeqIL-MQ · SVHN (Conv), CIFAR-10 (Conv), and Tiny ImageNet | test accuracy: 88.67(±.18) -> 82.76(±.87); 67.50(±.19) -> 59.72(±.48); 23.91(±.31) -> 18.22(±.29); 47.53(±.48) -> 39.58(±.46) | Hurt |
| replace MQ with Adam (ablation) | SeqIL-MQ · SVHN (FC) and CIFAR-10 (FC) | test accuracy: 84.51(±.17) -> 84.13(±.15); 57.70(±.21) -> 56.21(±.20) | No clear effect |
| use sequential rather than simultaneous inference (ablation) | IL-Adam on CIFAR-10 (FC) · CIFAR-10 | test performance | Helped |
| vary inference iterations from T=1 to T=5 (ablation) | SeqIL-Adam · CIFAR-10 | test performance: 1-5 | No clear effect |
| reduce inference iterations in simultaneous inference (ablation) | SimIL-Adam · CIFAR-10 | test performance | Hurt |
| use MQ with sequential inference instead of BP-SGD | SeqIL-MQ · SVHN, CIFAR-10, and Tiny ImageNet classification | loss reduction early in training | Helped |
| use sequential inference with MQ instead of BP-SGD | SeqIL-MQ · natural-image classification | convergence speed | Helped |
| use IL instead of BP | IL models · SVHN and CIFAR-10 autoencoders | reconstruction loss early in training | Helped |
| use IL-MQ or IL-Adam instead of BP algorithms | IL autoencoders · SVHN and CIFAR-10 autoencoders | reconstruction loss at convergence | No clear effect |
Surprising findings
- Plain SeqIL without an optimizer reduces loss faster than BP early in MLP training but fails to converge to good minima.
Limitations
- Sequential inference may be less parallelizable than standard IL, a possible drawback on some hardware.
- Theoretical results hold only under specific gamma settings and in limits (e.g., converged local errors, small beta).
- Probabilistic interpretation of the MQ optimizer is not fully analyzed.
Experimental setup
Datasets: SVHN CIFAR-10 Tiny ImageNet
Benchmarks: SVHN CIFAR-10 Tiny ImageNet
Metrics: test accuracy top-5 accuracy reconstruction loss convergence speed
Baselines: BP-SGD BP-Adam SeqIL-Adam SeqIL IL-Adam iPC
Builds on: inference learning implicit SGD Adam incremental predictive coding
Closest papers by meaning
- Divide-and-Conquer Predictive Coding: a structured Bayesian inference algorithm (NeurIPS 2024 · similarity 0.88)
- A Stable, Fast, and Fully Automatic Learning Algorithm for Predictive Coding Networks (ICLR 2024 · similarity 0.88)
- Understanding Sample Efficiency in Predictive Coding (NeurIPS 2026 · similarity 0.88)
- Predictive Coding beyond Correlations (ICML 2024 · similarity 0.88)
- Only Strict Saddles in the Energy Landscape of Predictive Coding Networks? (NeurIPS 2024 · similarity 0.87)
- $\mu$PC: Scaling Predictive Coding to 100+ Layer Networks (NeurIPS 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.