Research map / Architectures and efficiency / RNNs and normalization

$\mu$PC: Scaling Predictive Coding to 100+ Layer Networks

NeurIPS 2025 · Architectures and efficiency › RNNs and normalization · approach: Backprop-free credit assignment · 1 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes μPC, a Depth-μP-style reparameterisation of predictive coding networks that enables stable training of 100+ layer residual PCNs with zero-shot learning-rate transfer across width and depth.

Problem

Predictive coding networks (PCNs) struggle to train at large depth; standard parameterisation suffers from ill-conditioned inference landscapes and vanishing/exploding forward passes, preventing competitive large-scale local learning.

Approach

Reparameterise PC energy for residual networks with Depth-μP layer scalings (width- and depth-dependent premultipliers and init variances) so the forward pass stays stable at initialisation, enabling 128-layer PCNs and zero-shot transfer of weight and activity learning rates; also analyses conditioning of the activity Hessian and shows μPC approaches BP only when width >> depth.

Key results

• μPC trains fully connected residual PCNs up to 128 layers stably on MNIST, reaching about 98% test accuracy in 5 epochs, reported as 5x faster than the current benchmark.
• μPC enables zero-shot transfer of weight and activity learning rates across widths and depths (e.g., 16- to 128-layer Tanh networks).
• Theory: for linear ResNets, the equilibrated μPC energy converges to the MSE loss as depth/width ratio vanishes, but this correspondence breaks down with training at large width and depth.

Measured effects reported in the paper

Surprising findings

• Activity Hessian of standard PCNs and μPC becomes increasingly ill-conditioned with depth and training time, even though μPC keeps the forward pass stable.
• Applying Depth-μP learning-rate scaling by sqrt(NL) for Adam always led to non-trainable networks, so it was omitted.

Limitations

• Only simple classification tasks with fully connected residual networks; CIFAR10 performance is far from state of the art.
• Does not solve activity Hessian ill-conditioning, which grows during training.
• The BP correspondence theory holds only in a narrow regime (width much larger than depth) and is not proven for nonlinear networks.

Experimental setup

Datasets: MNIST Fashion-MNIST CIFAR10

Benchmarks: MNIST Fashion-MNIST CIFAR10

Metrics: test accuracy training loss condition number of activity Hessian

Baselines: standard parameterisation PC backpropagation with Depth-μP prior PC ResNet training (Pinchetti-style benchmarks)

Builds on: Depth-μP maximal update parameterisation (μP) predictive coding networks

Closest papers by meaning

• Only Strict Saddles in the Energy Landscape of Predictive Coding Networks? (NeurIPS 2024 · similarity 0.90)
• Training a Predictive Coding Network on ImageNet using Equilibrium Propagation (NeurIPS 2026 · similarity 0.89)
• Benchmarking Predictive Coding Networks -- Made Simple (ICLR 2025 · similarity 0.89)
• Local Loss Optimization in the Infinite Width: Stable Parameterization of Predictive Coding Networks and Target Propagation (ICLR 2025 · similarity 0.88)
• Understanding Sample Efficiency in Predictive Coding (NeurIPS 2026 · similarity 0.88)
• Tensor Programs VI: Feature Learning in Infinite Depth Neural Networks (ICLR 2024 · similarity 0.88)