Research map / Optimization / Optimizers
Accelerated training through iterative gradient propagation along the residual path
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces Highway-BP, a parallelizable iterative approximation of backpropagation that accumulates gradient estimates along residual paths and backpropagates them through layers in parallel via a prefix-scan, trading iterations for speed.
Problem
Backpropagation is inherently sequential and dominates training time for very deep sequential models (deep ResNets, Transformers, RNNs), limiting scalability.
Approach
Decomposes the gradient as a sum over all paths through residual connections and residual blocks, then computes a truncated sum with k iterations; each iteration backpropagates through all layers in parallel and propagates through residual paths with a parallel cumulative sum-product (prefix scan), so k controls the speed-accuracy tradeoff.
Key results
- ResNet110 on CIFAR10 matches backpropagation with only k=4 iterations while L=54.
- k=0 already reaches about 85% on CIFAR10, showing residual-path-only gradients carry most of the signal.
- Using optimal k, RNN training speedups between x2 and x3 with gains larger for longer sequences; on pixel-level CIFAR10 Highway-BP matches backpropagation with k=10.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use Highway-BP rather than fixed-point iteration | RNNs · RNN experiments | convergence speed | Helped |
| Use Highway-BP with k=10 rather than fixed-point iteration | 1-layer GRU · CIFAR10 pixel-level | accuracy | Helped |
| Use Highway-BP with k=4 rather than backpropagation | ResNet110 · CIFAR10 | performance | No clear effect |
| Set Highway-BP iterations to k=0 (ablation) | Pre-act ResNet32 · CIFAR10 | accuracy: 85% | Helped |
| Increase the number of Highway-BP iterations k (ablation) | deep sequential models · CIFAR10, ImageNet32, Wikitext103, MNLI | final performance | Helped |
| Increase Highway-BP iterations from low k (ablation) | GPT-2 · Wikitext103 | validation loss | Mixed |
| Set GPT-2 Highway-BP residual split to γ=0.2 instead of γ=0 (ablation) | GPT-2 · Wikitext103 | convergence | Helped |
| Use the standard GPT-2 residual split (γ=0) (ablation) | GPT-2 Highway-BP · Wikitext103 | convergence | Hurt |
| Set RoBERTa Highway-BP residual split to γ=0.8 (ablation) | RoBERTa · MNLI | convergence | Helped |
| Increase the number of Highway-BP iterations k (ablation) | RNNs · Wikitext103 char-level, Wikitext103, CIFAR10 pixel-level | performance | Mixed |
| Use Highway-BP with an optimal number of iterations | RNN models · RNN experiments | speedup: ×2 and ×3 | Helped |
Surprising findings
- Backpropagating only through the residual path (k=0) already gives reasonable performance, similar to boosting.
- Residual-path gradients keep a high contribution to the total norm throughout training, not only at the beginning.
Limitations
- Requires architecture-specific decomposition into g_i and r_i; GPT-2 and RoBERTa need tuned gamma mixing terms to converge.
- Distributed or large-scale implementation left for future work; prefix scan not yet optimized with custom kernels.
- Approximation quality depends on k and training stage; Transformer cosine similarity behavior varies.
Experimental setup
Datasets: CIFAR10 ImageNet32 Wikitext103 MNLI CIFAR10 pixel-level (Long Range Arena)
Benchmarks: CIFAR10 ImageNet32 Wikitext103 MNLI CIFAR10 pixel-level (Long Range Arena)
Metrics: test accuracy validation loss perplexity training step time cosine similarity to true gradient
Baselines: Backpropagation Fixed-point iteration (FPI)
Builds on: ResNet GPT-2 RoBERTa fixed-point iteration backpropagation CumSumProd prefix scan (Hillis and Steele)
Closest papers by meaning
- Backpropagation-Free Deep Learning with Recursive Local Representation Alignment (AAAI 2023 · similarity 0.89)
- Decoupled Parallel Backpropagation with Convergence Guarantee (ICML 2018 · similarity 0.87)
- Reducing Fine-Tuning Memory Overhead by Approximate and Memory-Sharing Backpropagation (ICML 2024 · similarity 0.87)
- PETRA: Parallel End-to-end Training with Reversible Architectures (ICLR 2025 · similarity 0.87)
- Stepping Forward on the Last Mile (NeurIPS 2024 · similarity 0.85)
- The Price of Locality: Why Forward-Forward Underperforms Backpropagation? (NeurIPS 2026 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.