Research map / Architectures and efficiency / RNNs and normalization
Stepping Forward on the Last Mile
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Investigates on-device training with fixed-point (16-bit weights, 8-bit perturbations and gradients) forward gradients via sign-m-SPSA and QZO-FF, showing near-backpropagation accuracy on few-shot vision/audio and cross-domain/OOD adaptation tasks with lower scratch memory.
Problem
Enabling on-device model adaptation on memory- and compute-constrained edge devices that only support fixed-point inference, where backpropagation's activation storage is prohibitive.
Approach
Formulates forward-gradient (SPSA-style two-forward-call) training entirely in fixed-point arithmetic, with a sign-based multi-sample estimator (sign-m-SPSA) to reduce gradient noise, plus optional momentum-guided sampling, sharpness-aware perturbation, sparse updates and kernel-wise normalization.
Key results
- On vision few-shot tasks, FF with 16w8a quantization stays within 5% of BP on 26 of 30 comparisons, e.g. ~77.4% vs 85.3% BP on CUB with ResNet12.
- On VWW cross-domain adaptation with linear probing, quantized FF reaches 87.30% from 48.50% zero-shot, a 0.63% gap to BP fp16.
- For full fine-tuning on ViT tiny, FF reduces scratch memory from 31.64MB to 11.43MB (2.8x) under fp16; with sparse update and fixed-point, 0.40MB scratch memory.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use quantized forward gradients (16w8a) instead of backpropagation | ViT tiny, linear probing · Visual Wake Word | accuracy: 87.87 -> 87.30% | Hurt |
| increase gradient averaging from m=1 to m=3 (ablation) | FF, fp16, D-VPT · Visual Wake Word | accuracy: 87.30 -> 88.94 | Helped |
| use 8-bit weights instead of 16-bit weights (ablation) | QZO-FF · Visual Wake Word | accuracy: 47.02; not converged | Hurt |
| prune 90% of trainable weights (ablation) | FF, ViT tiny · Cifar10-C | accuracy: 83.37 ($\pm$ 0.60) -> 83.34 ($\pm$ 0.59); 77.04 ($\pm$ 0.66) -> 77.11 ($\pm$ 0.68); 68.65 ($\pm$ 0.70) -> 68.63 ($\pm$ 0 | Hurt |
| quantize forward-gradient training to 16w8a (ablation) | FF, ViT tiny, LP 1 LN · Cifar10-C | accuracy: 83.37 ($\pm$ 0.60) -> 83.23 ($\pm$ 0.57); 77.04 ($\pm$ 0.66) -> 76.73 ($\pm$ 0.75); 68.65 ($\pm$ 0.70) -> 68.28 ($\pm$ 0 | Hurt |
| use quantized forward gradients (16w8a) rather than fp16 forward gradients (ablation) | FF, audio few-shot classification · ESC-50 and FSDKaggle18 | accuracy | No clear effect |
| replace normal perturbation sampling with Binomial sampling (ablation) | Forward gradient learning · Visual Wake Word | accuracy | No clear effect |
| use 16-bit rather than 8-bit weight quantization (ablation) | QZO-FF · Visual Wake Word | test error: 51.35% -> 12.74%; 52.98% -> 11.38% | Helped |
| use FF rather than BP | ViT tiny, LP 1 LN · Cifar10-C | accuracy: 83.75 ($\pm$ 0.67) -> 83.37 ($\pm$ 0.60); 77.88 ($\pm$ 0.85) -> 77.04 ($\pm$ 0.66); 70.03 ($\pm$ 1.20) -> 68.65 ($\pm$ 0 | Hurt |
| use FF rather than BP | AST, SimpleShot · ESC-50 | accuracy: 75.98 -> 79.70 | Helped |
Surprising findings
- 8-bit weight quantization (8w8a) fails to converge, so 16-bit weights are necessary to represent the tiny epsilon perturbation.
- Audio few-shot results show FF sometimes outperforms BP (e.g., AST SimpleShot ESC-50 79.70% vs 68.04%), suggesting the BP baseline is not always strongest.
Limitations
- 8-bit weights do not converge; 16-bit weights needed, limiting savings
- Accuracy gap to BP grows on hard corruptions and low-resolution inputs (up to ~7% on Cifar10-C hard), and FF converges more slowly
- Only classification tasks evaluated; training from scratch not guaranteed to converge
Experimental setup
Datasets: CUB Omniglot Cifar100_fs miniImageNet tieredImageNet ESC-50 FSDKaggle18 Visual Wake Words Cifar10-C ImageNet-1k Cifar10
Benchmarks: CUB Omniglot Cifar100_fs miniImageNet tieredImageNet ESC-50 FSDKaggle18 Visual Wake Words Cifar10-C
Metrics: accuracy test error scratch memory (MB)
Baselines: backpropagation (BP) fine-tuning BP linear probing zero-shot
Builds on: SPSA MeZO forward gradients (forward-mode AD) sign-SGD DeiT
Closest papers by meaning
- Accelerated On-Device Forward Neural Network Training with Module-Wise Descending Asynchronism (NeurIPS 2023 · similarity 0.88)
- Efficient On-Device Training via Gradient Filtering (CVPR 2023 · similarity 0.88)
- Accelerated training through iterative gradient propagation along the residual path (ICLR 2025 · similarity 0.85)
- FLOPS: Forward Learning with OPtimal Sampling (ICLR 2025 · similarity 0.85)
- An Empirical Study of Distributed Deep Learning Training on Edge (Student Abstract) (AAAI 2024 · similarity 0.85)
- Towards Exact Gradient-based Training on Analog In-memory Computing (NeurIPS 2024 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.