Research map / Architectures and efficiency / RNNs and normalization
Counter-Current Learning: A Biologically Plausible Dual Network Approach for Deep Learning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes counter-current learning (CCL), a biologically plausible dual-network scheme where a feedforward net processes inputs and a separate feedback net processes targets, with layerwise local losses between anti-parallel layer pairs and stop-gradient to avoid backward locking.
Problem
Error backpropagation is biologically implausible due to the weight transport problem, non-local credit assignment, and backward locking; existing target propagation methods are slow and still suffer locking.
Approach
A feedforward network maps inputs to outputs while a separate feedback network maps targets backward with its own weights; each pair of anti-parallel layers is aligned via a normalized cross-correlation-style local loss, with stop-gradient so each layer learns locally and forward/feedback passes can run without waiting on each other.
Key results
- On MLPs, CCL reaches 52.73% on CIFAR10 (best among methods, BP 50.03%) and 98.13% on MNIST vs BP 98.19%.
- On CNNs, CCL gets 56.29% on CIFAR100 vs BP 51.92%, but 82.94% on CIFAR10 vs BP 87.12%.
- Convolutional autoencoder on STL-10 reaches test MSE 0.0086 for CCL vs 0.0059 for BP.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use CCL instead of BP | MLP · MNIST | test accuracy: 98.19 ± 0.10 -> 98.13 ± 0.10 | No clear effect |
| Use CCL instead of BP | MLP · FashionMNIST | test accuracy: 89.58 ± 0.25 -> 88.58 ± 0.29 | Hurt |
| Use CCL instead of BP | MLP · CIFAR10 | test accuracy: 50.03 ± 0.31 -> 52.73 ± 0.59 | Helped |
| Use CCL instead of BP | MLP · CIFAR100 | test accuracy: 22.55 ± 0.19 -> 21.76 ± 0.22 | Hurt |
| Use CCL instead of BP | CNN · CIFAR10 | test accuracy: 87.12 -> 82.94 | Hurt |
| Use CCL instead of BP | CNN · CIFAR100 | test accuracy: 51.92 -> 56.29 | Helped |
| Use CCL instead of BP | convolutional autoencoder · STL-10 | mean square error: 0.0059± 0.0001 -> 0.0086± 0.0001 | Hurt |
| Use CCL instead of BP | 6-layer MLP · MNIST | FLOPs: 2.39 M -> 2.55 M | Hurt |
| Use CCL instead of BP | 4-layer MLP · CIFAR10 | FLOPs: 25.23 M -> 27.89 M | Hurt |
| Set the feedback network learning rate to zero (fixed random feedback weights) (ablation) | CCL MLP · classification tasks shown in the asymmetric-learning-rate ablation | test accuracy | No clear effect |
| Vary forward and feedback network learning rates asymmetrically (ablation) | CCL MLP · classification tasks shown in the asymmetric-learning-rate ablation | test accuracy | No clear effect |
Surprising findings
- CCL outperforms BP on CIFAR10 (MLP) and CIFAR100 (CNN) in the reported results.
- Robust learning is achieved even when the feedback network has zero learning rate (fixed random), which is reported to beat DRTP.
Limitations
- Limited theoretical understanding of learning dynamics, stability, and inductive biases.
- Requires extra stabilization tricks (activation normalization, feature-collapse regularizer, warm-up, flooding) and CNN results lag BP on CIFAR10.
- Backward locking only partially addressed; forward and feedback passes run sequentially in the runtime experiment.
Experimental setup
Datasets: MNIST FashionMNIST CIFAR10 CIFAR100 STL-10
Benchmarks: MNIST FashionMNIST CIFAR10 CIFAR100 STL-10
Metrics: test accuracy mean squared error FLOPs per sample centered kernel alignment
Baselines: BP FA DFA DRL L-DRL FW-DTP DRTP DTP
Builds on: difference target propagation fixed-weight difference target propagation direct random target projection feedback alignment gradient centralization
Closest papers by meaning
- Learning to Learn with Feedback and Local Plasticity (NeurIPS 2020 · similarity 0.86)
- Biological credit assignment through dynamic inversion of feedforward networks (NeurIPS 2020 · similarity 0.86)
- Beyond accuracy: generalization properties of bio-plausible temporal credit assignment rules (NeurIPS 2022 · similarity 0.86)
- Kernelized information bottleneck leads to biologically plausible 3-factor Hebbian learning in deep networks (NeurIPS 2020 · similarity 0.86)
- Feedback control guides credit assignment in recurrent neural networks (NeurIPS 2024 · similarity 0.84)
- Forward Learning with Top-Down Feedback: Empirical and Analytical Characterization (ICLR 2024 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.