Research map / Optimization / Optimizers
Can Forward Gradient Match Backpropagation?
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's title only (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use local-loss gradients (Local Guess) instead of isotropic random Gaussian/Rademacher guesses for the Forward Gradient direction | Forward Gradient with Global Target, ResNet-18 8 blocks · CIFAR-10 | test accuracy: up to ~40 points (e.g., 92.0 vs 50.0 weight perturbation) | Helped |
| Use local-loss gradients (Local Guess) instead of random guesses | Forward Gradient with Global Target, ResNet-18 8 blocks · ImageNet32 | test accuracy: e.g., 45.8 vs 4.9 | Helped |
| Weight perturbation instead of activity perturbation for supervised Local Guesses (ablation) | Forward Gradient with Global Target and Local Guess · CIFAR-10 8 and 16 blocks | test accuracy: e.g., 92.0 (weight) vs 90.4 (activity) for CNN, 8 blocks | Helped |
| Weight perturbation instead of activity perturbation for random Gaussian guesses with Local Target (ablation) | Forward Gradient with Local Target and random guesses · CIFAR-10 8 blocks | test accuracy: e.g., 52.1 (weight) vs 41.2 (activity) with CNN-style setup | Helped |
| Random NTK guesses (re-initialized auxiliary each iteration) instead of Gaussian guesses (ablation) | Forward Gradient with Global Target · CIFAR-10 8 blocks | test accuracy: slight improvement in most cases (e.g., 66.8 vs 50.0 for CNN weight) | Helped |
| Intermediate target (loss of block j+1) replacing the global end-to-end target (ablation) | Forward Gradient with Local Guess · CIFAR-10 8 blocks | test accuracy: 91.6 vs 92.0 (CNN weight), and lower than Global Target generally | Hurt |
| Projecting the global target onto the span of a batch of local guesses instead of a single local guess (ablation) | Forward Gradient with Global Target, activity perturbation · CIFAR-10 8 blocks | test accuracy: 92.3 vs 90.4 for single CNN local guess; improvement described as marginal | No clear effect |
| Wider networks (width factor k from 1 to 4) with random Gaussian guesses (ablation) | Forward Gradient with Global Target, Wide ResNet-18 · CIFAR-10 16 blocks | test accuracy: 48.7 at k=4 vs 50.0-ish at k=1 weight perturbation (16 blocks ResNet-18 Gaussian 50.0 is 8 block); random guess accuracy | Hurt |
| Wider networks (width factor k=4) with local CNN guesses (ablation) | Forward Gradient with Global Target and Local Guess · CIFAR-10 16 blocks | test accuracy: 93.3 at k=4 vs 85.5 at k=2 (weight, CNN) | Helped |
| Fixed randomly parameterized local loss as guess (Fixed NTK) instead of re-initialized NTK guess (ablation) | Forward Gradient with Global Target · CIFAR-10 8 blocks | test accuracy: Fixed NTK Linear 76.4 vs random NTK Linear 62.6 (weight) | No clear effect |
| Layerwise decoupling (one auxiliary per layer, 16 blocks) instead of residual-block decoupling (8 blocks) (ablation) | Forward Gradient with Global Target and Local Guess · CIFAR-10 | test accuracy: e.g., 88.7 (16 blocks MLP weight) vs 91.6 (8 blocks MLP weight) | Hurt |
| Using gradients from local losses only as target (Local Target) vs Global Target with Local Guess | Forward Gradient Local Guess · CIFAR-10 8 blocks | test accuracy: Local Target local learning 92.0 vs Global Target + Local Guess 92.0 (tie for CNN); Global Target generally better in ge | Hurt |
Closest papers by meaning
- Forward Learning with Top-Down Feedback: Empirical and Analytical Characterization (ICLR 2024 · similarity 0.86)
- Stepping Forward on the Last Mile (NeurIPS 2024 · similarity 0.85)
- One Forward is Enough for Neural Network Training via Likelihood Ratio Method (ICLR 2024 · similarity 0.85)
- FLOPS: Forward Learning with OPtimal Sampling (ICLR 2025 · similarity 0.85)
- Forward Learning of Graph Neural Networks (ICLR 2024 · similarity 0.84)
- Feature Averaging: An Implicit Bias of Gradient Descent Leading to Non-Robustness in Neural Networks (ICLR 2025 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.