Research map / Architectures and efficiency / Pruning and efficient tuning
One Forward is Enough for Neural Network Training via Likelihood Ratio Method
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes a unified likelihood ratio (ULR) method that estimates neural network gradients with a single forward pass via push-out LR, avoiding backpropagation, with variance reduction techniques and applications to computation-graph rearrangement and domain adaptation.
Problem
Backpropagation is the mainstream gradient method but its chain-rule recursion constrains network architecture design and training pipelines; alternatives such as forward-forward, evolution strategies, and perturbation methods are either unstable or costly.
Approach
Injects Gaussian noise into modules and applies the push-out likelihood ratio technique so each module's gradient is an expectation of the loss times a noise score, computable from one forward pass and localized per module, with layer-wise noise, antithetic sampling, and QMC for variance reduction.
Key results
- ULR achieves on average a 0.22% gap to BP on clean samples across architectures, while ES fails on ResNet-5 and GAT.
- ULR improves robustness on corrupted data by 9.53% on average compared with BP.
- On Office-31 domain adaptation with AlexNet, ULR improves average accuracy over BP (e.g., 1.2% higher) with a 1.1x training speedup (187 s vs 202 s).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use ULR instead of BP for domain adaptation | AlexNet with DRCN · Office-31 domain adaptation | classification accuracy: 1.2%↑ | Helped |
| Use ULR instead of BP for domain adaptation | AlexNet with DRCN · Office-31 domain adaptation | training speed: 1.1× speedups | Helped |
| Rearrange BPTT into ULR-decoupled subgraphs | RNN · Ag-News | accuracy: up to 4.9%↑ | Helped |
| Rearrange BPTT into ULR-decoupled subgraphs | RNN · Ag-News | speedup: up to 3.7× speedups | Helped |
| Use hybrid weight perturbation for the first two layers and logit perturbation for the others (ablation) | ULR on ResNet-5 · CIFAR-10 | gradient cosine similarity to BP | Helped |
| Remove layer-wise noise injection (ablation) | ULR on ResNet-5 · CIFAR-10 | gradient cosine similarity to BP | Hurt |
| Remove antithetic sampling (ablation) | ULR on ResNet-5 · CIFAR-10 | gradient cosine similarity to BP | Hurt |
| Use QMC instead of MC to generate noise (ablation) | ULR on ResNet-5 · CIFAR-10 | gradient cosine similarity to BP | No clear effect |
| Use ULR instead of BP | ULR across CNN, RNN, GNN, and SNN tasks · Classification tasks on CIFAR-10, MNIST, Fashion-MNIST, Ag-News, and Cora | clean-data classification accuracy: 0.22% | No clear effect |
| Use ULR instead of BP | ULR across CNN, RNN, GNN, and SNN tasks · Corrupted and noisy data | accuracy: 9.53% | Helped |
Surprising findings
- ULR training yields notably higher robustness on corrupted data than BP (about 9.5% on average).
Limitations
- Gradient estimates are noisy and require many noise copies in high-dimensional settings; variance grows with noise dimensionality.
- ES fails to optimize some models (ResNet-5, GAT), showing stability issues of zero-order methods.
Experimental setup
Datasets: CIFAR-10 MNIST Fashion-MNIST Ag-News Cora Office-31
Benchmarks: CIFAR-10 MNIST Fashion-MNIST Ag-News Cora Office-31
Metrics: classification accuracy robustness accuracy on corrupted data cosine similarity of gradients training time
Baselines: BP ES forward-forward HSIC bottleneck feedback alignment SPSA
Builds on: likelihood ratio method backpropagation forward-forward evolution strategy deep residual correction network
Closest papers by meaning
- Forward Learning with Top-Down Feedback: Empirical and Analytical Characterization (ICLR 2024 · similarity 0.87)
- Efficient Neural Network Training via Forward and Backward Propagation Sparsification (NeurIPS 2021 · similarity 0.86)
- Can Forward Gradient Match Backpropagation? (ICML 2023 · similarity 0.85)
- Stepping Forward on the Last Mile (NeurIPS 2024 · similarity 0.84)
- Accelerated training through iterative gradient propagation along the residual path (ICLR 2025 · similarity 0.84)
- Reducing Fine-Tuning Memory Overhead by Approximate and Memory-Sharing Backpropagation (ICML 2024 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.