Research map / Architectures and efficiency / RNNs and normalization
Pretraining with Random Noise for Fast and Robust Learning without Weight Transport
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Shows that pretraining a feedforward network on random Gaussian noise inputs with random labels, using feedback alignment, pre-aligns forward weights to fixed feedback weights and speeds up, improves, and regularizes subsequent data training without weight transport.
Problem
Feedback alignment avoids weight transport but needs large amounts of data to align forward and backward weights and underperforms backpropagation on harder tasks; the paper asks whether spontaneous-activity-like pretraining can pre-align weights and improve subsequent learning.
Approach
Before any real data, train the network on random Gaussian noise inputs paired with random labels using feedback alignment; this pre-aligns forward weights to fixed random feedback weights, lowers effective rank, and yields faster, more accurate, better-generalizing learning on subsequent data and tasks.
Key results
- On MNIST with a two-layer network, random noise pretraining improved FA accuracy from 97.26% to 97.76%, close to BP at 97.82%; on CIFAR-10 the gain was 3.04 points (50.54% to 53.58% vs BP 54.01%).
- Random noise pretraining reduced effective rank of forward weights and the generalization gap; out-of-distribution test accuracy on transformed MNIST and USPS was significantly higher than without pretraining.
- Gains in final accuracy grew with task complexity (0.49% on MNIST to 4.81% on STL-10) and the gap to BP widened with depth (up to 5 layers).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add random noise pretraining before data training (ablation) | Feedback alignment · MNIST | accuracy: $97.26 \pm 0.07$ -> $97.76 \pm 0.07$; $\blacktriangle 0.49 \pm 0.06$ | Helped |
| add random noise pretraining before data training (ablation) | Feedback alignment · Fashion-MNIST | accuracy: $87.47 \pm 0.25$ -> $88.26 \pm 0.07$; $\blacktriangle 0.79 \pm 0.31$ | Helped |
| add random noise pretraining before data training (ablation) | Feedback alignment · CIFAR-10 | accuracy: $50.54 \pm 0.22$ -> $53.58 \pm 0.12$; $\blacktriangle 3.04 \pm 0.20$ | Helped |
| add random noise pretraining before data training (ablation) | Feedback alignment · CIFAR-100 | accuracy: $20.17 \pm 0.30$ -> $24.45 \pm 0.10$; $\blacktriangle 4.28 \pm 0.38$ | Helped |
| add random noise pretraining before data training (ablation) | Feedback alignment · STL-10 | accuracy: $36.21 \pm 0.91$ -> $41.01 \pm 0.16$; $\blacktriangle 4.81 \pm 0.86$ | Helped |
| perform random noise training before data training rather than after it (ablation) | Feedback alignment · MNIST weight alignment | distance to synaptic feedback | Helped |
| add random noise pretraining before data training (ablation) | Feedback alignment · MNIST learning process | convergence speed (test-accuracy AUC) | Helped |
| add random noise pretraining before data training (ablation) | Feedback alignment · MNIST | generalization error | Helped |
| add random noise pretraining before data training (ablation) | Feedback alignment · Translated, rotated, and scaled MNIST | test accuracy | Helped |
| add random noise pretraining before data training (ablation) | Feedback alignment · USPS | test accuracy | Helped |
| add random noise pretraining (ablation) | Feedback alignment · MNIST, Fashion-MNIST, and Kuzushiji-MNIST adaptation | adaptation speed | Helped |
| add random noise pretraining (ablation) | Feedback alignment · CIFAR-10 | total training time to convergence | Helped |
Surprising findings
- Training on pure random noise with random labels reduced loss and aligned weights, even when inputs and labels were randomly paired.
- Performing data training before random pretraining prevented subsequent random training from moving weights toward feedback.
Limitations
- Only fully connected feedforward networks were studied; convolutional architectures not tested.
- Performance gap with backpropagation widens as depth increases.
- Experiments use small data subsets (5000 training samples) and small MLPs.
Experimental setup
Datasets: MNIST Fashion-MNIST CIFAR-10 CIFAR-100 STL-10 USPS Kuzushiji-MNIST random Gaussian noise
Benchmarks: MNIST Fashion-MNIST CIFAR-10 CIFAR-100 STL-10 USPS Kuzushiji-MNIST
Metrics: test accuracy AUC of test accuracy (convergence speed) alignment angle effective rank generalization error meta-loss
Baselines: backpropagation feedback alignment without random pretraining
Builds on: feedback alignment backpropagation meta-learning loss
Closest papers by meaning
- One-Time Soft Alignment Enables Resilient Learning without Weight Transport (NeurIPS 2026 · similarity 0.87)
- Forward Learning with Top-Down Feedback: Empirical and Analytical Characterization (ICLR 2024 · similarity 0.87)
- Understanding and Mitigating the Label Noise in Pre-training on Downstream Tasks (ICLR 2024 · similarity 0.86)
- Overcoming Rank Collapse in Feedback Alignment (NeurIPS 2026 · similarity 0.85)
- Implicit Regularization in Feedback Alignment Learning Mechanisms for Neural Networks (ICML 2024 · similarity 0.85)
- Feedback control guides credit assignment in recurrent neural networks (NeurIPS 2024 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.