Research map / Architectures and efficiency / RNNs and normalization
Only Strict Saddles in the Energy Landscape of Predictive Coding Networks?
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Shows that at PC inference equilibrium the energy of deep linear networks is a weight-rescaled MSE loss, and that the origin and zero-rank saddles become strict in this energy, conjecturing all its saddles are strict.
Problem
Predictive coding (PC) is claimed to converge faster than backprop, but the impact of its inference procedure on the learning landscape is not theoretically understood; in particular, how saddle points of the MSE loss (e.g. the origin in deep linear networks, which causes vanishing gradients) differ
Approach
At the inference equilibrium, the PC energy of a deep linear network equals the MSE loss rescaled by a weight-dependent matrix S = I + sum_l (W_{L:l})(W_{L:l})^T. This rescaling makes many highly degenerate non-strict saddles of the loss (including the origin) strict, so gradient descent escapes them much faster.
Key results
- The equilibrated PC energy of deep linear networks matches a rescaled MSE loss, with excellent numerical agreement with theory.
- The origin saddle of the equilibrated energy has a negative eigenvalue for any depth H>=1, so it is strict, whereas the MSE loss Hessian at the origin is zero for H>1.
- On non-linear networks initialised near the origin (sigma=5e-3, lr=1e-3), PC escapes the origin much faster than BP with SGD and shows no vanishing gradients.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use the equilibrated predictive-coding energy instead of the MSE loss | Deep linear networks at the origin saddle · Deep linear networks | saddle strictness | Helped |
| Use the equilibrated predictive-coding energy instead of the MSE loss | Deep linear networks near the origin · Toy linear networks | escape speed | Helped |
| Use predictive coding instead of backpropagation | Linear and non-linear networks, including convolutional architectures · MNIST, Fashion-MNIST, and CIFAR-10 | escape speed | Helped |
| Use predictive coding instead of backpropagation | Networks initialized near higher-rank saddles · Matrix completion (rank-3 target) | escape speed | Helped |
| Use the equilibrated predictive-coding energy instead of the MSE loss | Deep linear networks at zero-rank critical points · Deep linear networks | saddle strictness | Helped |
| Use predictive coding with inference at equilibrium | Non-linear networks · MNIST, Fashion-MNIST, and CIFAR-10 | vanishing gradients | Helped |
Surprising findings
- Prior claims that the MSE loss equals the output energy at equilibrium are incorrect; this holds only at the feedforward pass.
- The claim of universally faster convergence of PC learning may be overstated.
Limitations
- Strictness of energy saddles holds only at the exact inference equilibrium.
- Theoretical results cover deep linear networks; non-linear results are empirical only.
- Higher-rank saddles are supported only experimentally; the conjecture that all energy saddles are strict is unproven.
Experimental setup
Datasets: MNIST MNIST-1D Fashion-MNIST CIFAR-10 synthetic toy data synthetic rank-3 matrix
Benchmarks: MNIST MNIST-1D Fashion-MNIST CIFAR-10 matrix completion
Metrics: training loss weight gradient norm Hessian eigenspectrum escape time from saddle
Baselines: backpropagation with SGD
Builds on: predictive coding deep linear networks saddle analysis (Saxe et al.) strict saddle theory (Ge et al.) Kawaguchi 2016 deep linear loss landscape
Closest papers by meaning
- $\mu$PC: Scaling Predictive Coding to 100+ Layer Networks (NeurIPS 2025 · similarity 0.90)
- Training a Predictive Coding Network on ImageNet using Equilibrium Propagation (NeurIPS 2026 · similarity 0.89)
- Benchmarking Predictive Coding Networks -- Made Simple (ICLR 2025 · similarity 0.88)
- Augmented Lagrangian Predictive Coding (NeurIPS 2026 · similarity 0.88)
- Understanding Sample Efficiency in Predictive Coding (NeurIPS 2026 · similarity 0.88)
- Understanding and Improving Optimization in Predictive Coding Networks (AAAI 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.