Research map / Architectures and efficiency / Pruning and efficient tuning
Efficient Backpropagation with Variance Controlled Adaptive Sampling
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes VCAS, an unbiased adaptive sampling method that speeds up backpropagation by sampling data (activation gradients) and tokens (weight gradients) while controlling added variance via online-learned sample ratios.
Problem
Sampling-based training acceleration by skipping computations in backpropagation can introduce approximation error that hurts accuracy and convergence across tasks, and the appropriate sample ratio varies by task and training phase.
Approach
Compute an unbiased approximate stochastic gradient with layerwise importance sampling over the data dimension for activation gradients and leverage-score sampling over tokens for weight gradients, and adapt the sample ratios online by zeroth-order updates so the extra variance stays a small fraction (tau ≪ 1) of the SGD variance.
Key results
- VCAS reduces BP FLOPs by 41.56% on BERT-base MNLI fine-tuning while keeping eval accuracy at 84.23% vs 84.33% exact.
- On ViT-large ImageNet-1k fine-tuning, VCAS achieves 49.58% FLOPs reduction with 82.27% accuracy vs 82.04% exact, while SB and UB reach 44.44% reduction.
- Wall-clock time on BERT-large MNLI fine-tuning drops about 19.00% for VCAS versus 21.14% (SB) and 22.12% (UB), with VCAS keeping the training loss closest to exact.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use VCAS instead of UB | BERT-base · MNLI-m | eval accuracy (%): 83.82 -> 84.23 | Helped |
| Use VCAS instead of UB | BERT-large · MNLI-m | eval accuracy (%): 86.09 -> 86.63 | Helped |
| Use VCAS instead of UB | ViT-base · ImageNet-1k | eval accuracy (%): 82.28 -> 82.27 | No clear effect |
| Use VCAS instead of UB | ViT-large · ImageNet-1k | wall-clock time reduction (%): 19.83 -> 21.06 | Helped |
| Use VCAS instead of exact backpropagation | BERT-large · MNLI | FLOPs reduction (%): 44.17 | Helped |
| Use VCAS instead of exact backpropagation | ViT-large · ImageNet-1k | FLOPs reduction (%): 49.58 | Helped |
| Use VCAS with activation sampling only instead of exact training | WideResNet-18 (widen factor 4) · ImageNet pretraining | eval accuracy (%): 75.96 -> 75.86 | No clear effect |
| Jointly sample activations and weights instead of sampling either alone (ablation) | VCAS · BERT-base finetuning on MNLI | FLOPs reduction | Helped |
| Increase variance threshold τ from 0.025 to 0.5 (ablation) | VCAS · BERT-base finetuning on SST-2 | FLOPs reduction (%): 48.28 -> 52.71 | Helped |
| Increase variance threshold τ from 0.025 to 0.5 (ablation) | VCAS · BERT-base finetuning on SST-2 | eval accuracy (%): 93.04 -> 92.18 | Hurt |
| Increase adaptation frequency F from 100 to 1000 (ablation) | VCAS · BERT-base finetuning on SST-2 | FLOPs reduction (%): 48.28 -> 31.03 | Hurt |
| Vary Monte-Carlo repetitions M from 2 to 10 (ablation) | VCAS · BERT-base finetuning on SST-2 and MNLI | empirical variance | No clear effect |
Surprising findings
- Fixed-ratio baselines SB and UB diverge in training loss (e.g., SB reaches 0.3833 vs 0.2372 exact on MNLI), whereas VCAS tracks the exact trajectory.
- Wall-clock time reduction is smaller than FLOPs reduction (about 19% time for 44% FLOPs on BERT-large MNLI).
Limitations
- The weight sampler (SampleW) is designed only for linear layers and is not usable for convolutions, so CNNs need an activation-only variant.
- Not suitable for small models with low gradient variance or small datasets with too few steps for the adaptation to converge.
- Focuses on mirroring exact training, so it does not explore alternative convergence trajectories that might yield better results.
Experimental setup
Datasets: GLUE (MNLI, QQP, QNLI, SST-2, CoLA, STSB, MRPC, RTE) C4 CIFAR10 CIFAR100 ImageNet-1k
Benchmarks: GLUE (MNLI, QQP, QNLI, SST-2) C4 pretraining (GLUE average) CIFAR10 CIFAR100 ImageNet-1k
Metrics: final train loss eval accuracy FLOPs reduction wall-clock time reduction
Baselines: SB UB exact training
Builds on: selective backprop upper bound importance sampling (UB) leverage score sampling
Closest papers by meaning
- Overcoming Rank Collapse in Feedback Alignment (NeurIPS 2026 · similarity 0.86)
- Reducing Fine-Tuning Memory Overhead by Approximate and Memory-Sharing Backpropagation (ICML 2024 · similarity 0.85)
- FLOPS: Forward Learning with OPtimal Sampling (ICLR 2025 · similarity 0.85)
- Forward Learning with Top-Down Feedback: Empirical and Analytical Characterization (ICLR 2024 · similarity 0.84)
- Reparameterized Importance Sampling for Robust Variational Bayesian Neural Networks (ICML 2024 · similarity 0.84)
- Repeated Random Sampling for Minimizing the Time-to-Accuracy of Learning (ICLR 2024 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.