Research map / Architectures and efficiency / RNNs and normalization
Two Tales of Single-Phase Contrastive Hebbian Learning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Gives a rigorous derivation of dual propagation (DP) via relaxed optimal value reformulations and proposes DP^T, a Lagrangian-based variant that stays stable under asymmetric nudging.
Problem
Dual propagation, a single-phase local learning method for neural networks with dyadic neurons, has an objective lacking rigorous foundation and its inference only converges under symmetric nudging (alpha=1/2), which is restrictive for noisy analog/biological hardware.
Approach
Derives DP objective from repeated relaxations of the optimal value reformulation (SPROVR and AROVR), shows a link to adversarial robustness, and proposes an adjoint-state/Lagrangian variant (DP^T) whose fixed-point updates remain stable for asymmetric nudging.
Key results
- On MNIST with an MLP, DP with alpha=0 reaches 98.46% accuracy with Lipschitz estimate 29.9 versus 95.85% and 522 for alpha=1, showing lower alpha yields smaller Lipschitz constants for DP.
- DP^T with alpha=0 and 30 inference iterations keeps 98.4% MNIST accuracy, whereas original DP diverges (X) under asymmetric nudging with multiple iterations.
- On ImageNet32x32 with DP^T (alpha=1/2, beta=0.01) the model reaches 41.59% top-1 and 65.63% top-5, on par with reported DP and BP results.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| decrease α from 1 to 0 (ablation) | DP · MNIST test | accuracy: 95.85\!\pm\!0.68 -> 98.46\!\pm\!0.07 | Helped |
| decrease α from 1 to 0 (ablation) | DP · FashionMNIST test | accuracy: 83.36\!\pm\! 0.74 -> 89.16\!\pm\!0.43 | Helped |
| decrease α from 1 to 0 (ablation) | DP · MNIST test | Lipschitz estimate (L̂): 522\!\pm\!288~ -> 29.9\!\pm\!4.2~~ | Helped |
| decrease α from 1 to 0 (ablation) | DP · FashionMNIST test | Lipschitz estimate (L̂): 157.7\!\pm\!57.8 -> 19.97\!\pm\!3.9~~ | Helped |
| decrease α (ablation) | DPᵀ · MNIST and FashionMNIST test | accuracy | No clear effect |
| add input-layer robustness to DP with α=0 (ablation) | DP, MNIST test · MNIST test | Lipschitz estimate (L̂): 29.9\!\pm\!4.2~~ -> 15.87\!\pm\!6.34 | Helped |
| add input-layer robustness to DP with α=0 (ablation) | DP, MNIST test · MNIST test | accuracy: 98.46\!\pm\!0.07 -> 98.43\!\pm\!0.11 | Hurt |
| add input-layer robustness to DP with α=0 (ablation) | DP, FashionMNIST test · FashionMNIST test | Lipschitz estimate (L̂): 19.97\!\pm\!3.9~~ -> 8.05\!\pm\!1.03 | Helped |
| add input-layer robustness to DP with α=0 (ablation) | DP, FashionMNIST test · FashionMNIST test | accuracy: 89.16\!\pm\!0.43 -> 88.29\!\pm\!0.79 | Hurt |
| increase inference iterations from 1 to 30 (ablation) | DPᵀ with asymmetric α · MNIST test | training convergence | Helped |
| use original DP instead of DPᵀ with asymmetric α and 30 inference iterations (ablation) | DP, α∈{0,1} · MNIST | training convergence | Hurt |
| vary β from 0.01 to 1.0 (ablation) | DPᵀ, α=0 · CIFAR10 | top-1 accuracy: 92.26 \!\pm\! 0.22 -> 91.34 \!\pm\! 0.16 | Hurt |
Surprising findings
- Original dual propagation diverges under asymmetric nudging when run for repeated inference iterations, while the adjoint variant remains stable.
- Lower alpha in DP reduces the Lipschitz estimate and also improves test accuracy on MNIST and Fashion-MNIST.
Limitations
- Asymmetric nudging requires slow, stabilized inference iterations on digital hardware, adding significant computational cost.
- The weight transport problem is not addressed; DP and DP^T rely on symmetric feedback weights.
- Theoretical analysis of fixed-point updates relies on local linearity and idealized quadratic assumptions.
Experimental setup
Datasets: MNIST Fashion-MNIST CIFAR10 CIFAR100 ImageNet32x32
Benchmarks: MNIST Fashion-MNIST CIFAR10 CIFAR100 ImageNet32x32
Metrics: test accuracy top-1 accuracy top-5 accuracy spectral norm Lipschitz estimate gradient angle
Baselines: backpropagation dual propagation FGSM adversarial training LPOM
Builds on: dual propagation equilibrium propagation contrastive hebbian learning LPOM coupled learning
Closest papers by meaning
- Improving equilibrium propagation without weight symmetry through Jacobian homeostasis (ICLR 2024 · similarity 0.87)
- Unconditional stability of a recurrent neural circuit implementing divisive normalization (NeurIPS 2024 · similarity 0.86)
- Spike-timing-dependent Hebbian learning as noisy gradient descent (NeurIPS 2025 · similarity 0.86)
- Rethinking Hebbian Principle: Low-Dimensional Structural Projection for Unsupervised Learning (NeurIPS 2025 · similarity 0.85)
- Hebbian Learning based Orthogonal Projection for Continual Learning of Spiking Neural Networks (ICLR 2024 · similarity 0.85)
- Dendritic Localized Learning: Toward Biologically Plausible Algorithm (ICML 2025 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.