Research map / Optimization / Implicit bias and generalization
Local Loss Optimization in the Infinite Width: Stable Parameterization of Predictive Coding Networks and Target Propagation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Derives maximal update parameterization (µP) for predictive coding (PC) and target propagation (TP) in the infinite-width limit, verifies learning-rate transfer across widths, and analyzes linear networks to show PC gradients interpolate between GD and Gauss-Newton-like targets.
Problem
Local learning rules such as predictive coding and target propagation have many extra hyperparameters and unstable training, and lack theoretical scaling guidance that would let hyperparameters transfer across network widths.
Approach
Extends the µP framework (infinitesimal one-step perturbation analysis) to local loss optimization: derives width scalings for PC (including the inference step size of the last layer, gamma_L, as a scaling variable and without the fixed prediction assumption) and for TP/DTP with linear feedback networks; for linear PC gives closed-form fixed points showing gradients interpolate between GD and GNT
Key results
- µP for PC with single-shot sequential inference enables learning-rate transfer across widths on a 3-layer MLP on FashionMNIST, even without the fixed prediction assumption.
- For linear networks, PC's gradient reduces to first-order GD when gamma_L=Theta(1) and the last-layer dimension is 1 or the width is large, while gamma_L scaling toward GNT-like behavior with bar-gamma_L=-1.
- For TP/DTP with a linear feedback network, µP requires b_L=1/2 and the kernel regime disappears, so feature learning is favored.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use the derived μP instead of standard parameterization | Predictive Coding · FashionMNIST | test accuracy | Helped |
| Use the derived μP without fixed prediction assumption | Predictive Coding · FashionMNIST | learning-rate transfer across widths | Helped |
| Set the last-layer inference step-size exponent to -1 | Predictive Coding with 100 inference iterations · FashionMNIST | learning-rate transfer across widths | Helped |
| Set the last-layer inference step-size exponent to -1 | Predictive Coding without forward initialization · FashionMNIST | test accuracy | Helped |
| Use a larger inference step size | Predictive Coding · hyperparameter transfer across widths | hyperparameter transfer | Helped |
| Use the derived μP instead of standard parameterization | Target Propagation · learning across widths | hyperparameter transfer | Helped |
| Set the last-layer inference step-size exponent to 0 instead of -1 (ablation) | Predictive Coding with 100 inference iterations · FashionMNIST | inference stability across widths | Hurt |
Surprising findings
- For TP, the kernel regime does not exist: the standard last-layer scaling b_L=1/2 that yields a kernel regime for gradient methods instead induces feature learning.
- Despite theoretical differences from GNT, PC's gradient in wide networks becomes close to first-order GD in the standard setting.
Limitations
- Derivation uses only a one-step (infinitesimal) update and linear networks for the multi-step inference analysis.
- TP analysis assumes a linear feedback network activation.
- Experiments are small (3-layer MLP/CNN, VGG5 on FashionMNIST), not large-scale.
Experimental setup
Datasets: FashionMNIST
Benchmarks: FashionMNIST
Metrics: test accuracy learning-rate transfer across widths cosine similarity of gradients
Baselines: standard parameterization (SP) SGD µP GNT µP backpropagation
Builds on: maximal update parameterization (µP) predictive coding target propagation Gauss-Newton target tensor programs
Closest papers by meaning
- $\mu$PC: Scaling Predictive Coding to 100+ Layer Networks (NeurIPS 2025 · similarity 0.88)
- On the Surprising Effectiveness of Large Learning Rates under Standard Width Scaling (NeurIPS 2025 · similarity 0.88)
- Towards Interpretable Deep Local Learning with Successive Gradient Reconciliation (ICML 2024 · similarity 0.88)
- Depthwise Hyperparameter Transfer in Residual Networks: Dynamics and Scaling Limit (ICLR 2024 · similarity 0.87)
- On the Parameterization of Second-Order Optimization Effective towards the Infinite Width (ICLR 2024 · similarity 0.87)
- Global Convergence and Rich Feature Learning in $L$-Layer Infinite-Width Neural Networks under $\mu$ Parametrization (ICML 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.