Research map / Architectures and efficiency / RNNs and normalization
Brain network science modelling of sparse neural networks enables Transformers and LLMs to perform as fully connected
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes CHTs and CHTss, brain-inspired dynamic sparse training methods with soft link sampling, a GPU-friendly node-based link predictor, a bipartite receptive field initialization, and sigmoid density decay, reaching dense-model performance at high sparsity in MLPs, Transformers and LLaMA.
Problem
Dynamic sparse training (DST) methods struggle to maintain peak performance at high sparsity; the existing Cannistraci-Hebb training (CHT) regrowth is computationally expensive (O(N·d^3)) and rigidly selects top link-prediction scores, causing epitopological local minima.
Approach
Replace deterministic top-k link removal and regrowth with multinomial soft sampling, use a node-based CH2-L3 link predictor computable by matrix multiplication (O(N^3)), initialize with a bipartite receptive field (BRF) topology, and decay density with a sigmoid schedule.
Key results
- CHTs at 99% sparsity reaches 69.97% accuracy on CIFAR10 MLP vs 62.85% for fully connected (+11.33%), with CHTss at 71.29%.
- CHTss at 5% density (95% sparsity) surpasses the fully connected Transformer on Multi30k (32.03 BLEU vs 31.38 FC) and IWSLT (24.51 vs 24.48).
- On OpenWebText at 70% sparsity, CHTs on LLaMA-1B achieves perplexity 14.53 versus 14.62 for the fully connected model.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use CHTss instead of FC at 99% sparsity | MLP · CIFAR10 | accuracy: 62.85 $\pm$ 0.16 -> 71.29 $\pm$ 0.14 | Helped |
| Add sigmoid gradual density decay to CHTs | Transformer · Multi30k en-de (95% sparsity) | BLEU: 28.94 ± 0.57 -> 32.03 ± 0.29 | Helped |
| Use CHTss instead of FC at 95% sparsity | Transformer · IWSLT en-de | BLEU: 24.48 ± 0.30 -> 24.51 ± 0.02 | No clear effect |
| Use CHTs instead of FC at 70% sparsity | LLaMA-1B · OpenWebText | validation perplexity: 14.62 -> 14.53* | Helped |
| Replace path-based CH3-L3p regrowth with node-based CH2-L3n (ablation) | CHTs · MNIST, Fashion MNIST, and EMNIST | accuracy: 98.81 $\pm$ 0.04* -> 98.76 $\pm$ 0.05; 90.93 $\pm$ 0.03* -> 90.67 $\pm$ 0.05; 87.61 $\pm$ 0.07* -> 87.82 $\pm$ 0.04* | Mixed |
| Use relative importance rather than weight magnitude for link removal (ablation) | CHTs · Fashion MNIST | accuracy | Helped |
| Use weight magnitude rather than relative importance for link removal (ablation) | CHTs · MNIST and EMNIST | accuracy | Helped |
| Use BRF initialization instead of BSW initialization (ablation) | Sparse network initialization · MLP classification | win rate | Helped |
| Use equal partitioning and degree-based neuron resorting for BSF initialization (ablation) | BSF initialization · MLP classification | win rate | Helped |
| Use equal partitioning and degree-based neuron resorting for BSF initialization rather than BSW (ablation) | Sparse network initialization · MLP classification | win rate | Hurt |
| Use sigmoid density decay rather than cubic density decay (ablation) | CHTss · Transformer machine translation and language modeling | performance | Helped |
| Use soft rather than deterministic regrowth (ablation) | CHTs · OpenWebText (LLaMA-60M) | validation perplexity | Helped |
Surprising findings
- Sparse CHTs with 70% sparsity on LLaMA-1B outperforms the dense counterpart in perplexity.
- RigL and GraNet perform poorly under bfloat16 precision on LLaMA, attributed to imprecise gradients for regrowth, while gradient-free CHT variants are more robust.
Limitations
- Unstructured sparse training lacks widely adopted hardware, so no real training speedup is demonstrated.
- Large-model results are reported as single seeds for LLaMA experiments; three-seed runs deferred to revision.
- Soft-sampling temperature is hand-set rather than adapted per layer or epoch.
Experimental setup
Datasets: MNIST Fashion MNIST EMNIST CIFAR10 Multi30k en-de IWSLT14 en-de WMT17 en-de OpenWebText GLUE SuperGLUE
Benchmarks: MNIST Fashion MNIST EMNIST CIFAR10 Multi30k IWSLT14 WMT17 OpenWebText
Metrics: accuracy BLEU validation perplexity active neuron percolation rate ITOP rate
Baselines: FC SET RigL CHT MEST GMP GraNet CHTs CHTss
Builds on: Cannistraci-Hebb training (CHT) SET RigL GraNet MEST
Closest papers by meaning
- Improving the Sparse Structure Learning of Spiking Neural Networks from the View of Compression Efficiency (ICLR 2025 · similarity 0.86)
- The Graphon Limit Hypothesis: Understanding Neural Network Pruning via Infinite Width Analysis (NeurIPS 2025 · similarity 0.85)
- Dynamic Sparse No Training: Training-Free Fine-tuning for Sparse LLMs (ICLR 2024 · similarity 0.85)
- Return of ChebNet: Understanding and Improving an Overlooked GNN on Long Range Tasks (NeurIPS 2025 · similarity 0.85)
- Leveraging Low-Rank and Sparse Recurrent Connectivity for Robust Closed-Loop Control (ICLR 2024 · similarity 0.85)
- The Iterative Optimal Brain Surgeon: Faster Sparse Recovery by Leveraging Second-Order Information (NeurIPS 2024 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.