Research map / Architectures and efficiency / RNNs and normalization
Rethinking Hebbian Principle: Low-Dimensional Structural Projection for Unsupervised Learning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes SPHeRe, a feedforward, block-wise Hebbian unsupervised learning method that simplifies Oja's-rule equivalent loss into a Gram-matrix similarity objective computed on a low-dimensional auxiliary projection, plus an orthogonality loss, achieving SOTA among unsupervised synaptic plasticity methods on CIFAR-10/100 and Tiny-ImageNet.
Problem
Hebbian learning suffers from unconstrained weight updates and lacks feedback mediation, limiting its scaling to deep nonlinear networks and complex tasks; existing Hebbian methods lack a clear optimizable objective.
Approach
Simplify the equivalent loss of Oja's rule into a Gram-matrix matching loss ||YY^T - XX^T||^2 applied to a low-dimensional output of a lightweight auxiliary nonlinear projection on top of each block, combined with a feature orthogonality loss, enabling purely feedforward greedy layer-wise training.
Key results
- SPHeRe reaches 81.11% on CIFAR-10, 56.79% on CIFAR-100 and 40.33% on Tiny-ImageNet, above SoftHebb's reported 80.3% on CIFAR-10 and self-run 34.12% on Tiny-ImageNet.
- SPHeRe-EWC achieves 76.53% on Split-CIFAR100 and 67.05% on Split-TinyImageNet in continual learning.
- Transfer from Tiny-ImageNet to CIFAR-10 gives 80.03%, close to the 81.11% direct-training result.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use SPHeRe instead of SoftHebb | Unsupervised synaptic-plasticity classification · CIFAR-10 | accuracy: 80.3 -> 81.11 ± 0.11 | Helped |
| Use SPHeRe instead of SoftHebb | Unsupervised synaptic-plasticity classification · CIFAR-100 | accuracy: 56.0 -> 56.79 ± 0.69 | Helped |
| Use SPHeRe instead of SoftHebb | Unsupervised synaptic-plasticity classification · Tiny-ImageNet | accuracy: 34.12 -> 40.33 ± 0.24 | Helped |
| Combine SPHeRe with EWC | SPHeRe continual learning · Split-CIFAR100 and Split-TinyImageNet | accuracy: 72.72± 1.14 -> 76.53± 0.64; 63.46± 0.79 -> 67.05± 1.16 | Helped |
| Transfer features from Tiny-ImageNet to CIFAR-10 | SPHeRe · CIFAR-10 | test accuracy: 81.11 -> 80.03; $-1.08\%$ | Hurt |
| Transfer features from CIFAR-10 to Tiny-ImageNet | SPHeRe · Tiny-ImageNet | test accuracy: 40.33 -> 37.7; $-2.63\%$ | Hurt |
| Use SPHeRe features instead of BP features | CIFAR-10 reconstruction encoder · CIFAR-10 | reconstruction loss: 1.68 -> 1.62 | Helped |
| Use only the original Oja loss rather than the full SPHeRe combination (ablation) | Unsupervised classifier · CIFAR-10 | accuracy: 81.18 -> 73.8 | Hurt |
| Add the auxiliary block to Oja loss (ablation) | Oja-loss classifier · CIFAR-10 | accuracy: 73.8 -> 76.2 | Helped |
| Add orthogonality loss to Oja loss with auxiliary block (ablation) | Oja-loss classifier with auxiliary block · CIFAR-10 | accuracy: 76.2 -> 76.3 | No clear effect |
| Add the auxiliary block to SPHeRe loss (ablation) | SPHeRe-loss classifier · CIFAR-10 | accuracy: 65.4 -> 78.9 | Helped |
| Combine SPHeRe loss, orthogonality loss, and auxiliary block (ablation) | Unsupervised classifier · CIFAR-10 | accuracy: 80.7 -> 81.18 | Helped |
Surprising findings
- Performance gains diminish as the number of convolutional layers increases, attributed to noise accumulation in purely local learning.
- SoftHebb training reduced accuracy in the spiking network setting relative to random weights.
Limitations
- Gains diminish as depth increases due to noise accumulation in purely local learning.
- Performance is sensitive to activation function choice.
- Weaker on animal classes with fine textures in t-SNE analysis.
Experimental setup
Datasets: CIFAR-10 CIFAR-100 Tiny-ImageNet Split-CIFAR100 Split-TinyImageNet
Benchmarks: CIFAR-10 CIFAR-100 Tiny-ImageNet Split-CIFAR100 Split-TinyImageNet
Metrics: test accuracy reconstruction MSE
Baselines: D-CSNN Hard WTA SoftHebb EWC HAT OWM GPM SimCLR BP
Builds on: Oja's rule SoftHebb similarity matching EWC
Closest papers by meaning
- Hebbian Learning based Orthogonal Projection for Continual Learning of Spiking Neural Networks (ICLR 2024 · similarity 0.90)
- Spike-timing-dependent Hebbian learning as noisy gradient descent (NeurIPS 2025 · similarity 0.86)
- Two Tales of Single-Phase Contrastive Hebbian Learning (ICML 2024 · similarity 0.85)
- What Do Hebbian Learners Learn? Reduction Axioms for Iterated Hebbian Learning (AAAI 2024 · similarity 0.85)
- Multi-View Incremental Learning with Structured Hebbian Plasticity for Enhanced Fusion Efficiency (AAAI 2025 · similarity 0.84)
- Learning Successor Features with Distributed Hebbian Temporal Memory (ICLR 2025 · similarity 0.83)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.