Research map / Architectures and efficiency / Quantization
Few-bit Backward: Quantized Gradients of Activation Functions for Memory Footprint Reduction
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's title only (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use 3-bit rather than vanilla GELU backward gradients | RoBERTa-base · GLUE | task metric | No clear effect |
| Use 1- or 2-bit rather than vanilla GELU backward gradients | RoBERTa-base · GLUE and ruDALL-E Malevich training | task performance | Hurt |
| Quantize activation backward gradients to 3 bits | RoBERTa-base · GLUE MRPC fine-tuning | peak memory (GiB): 11.34 -> 9.78; 13.8% | Helped |
| Quantize activation backward gradients to 3 bits | RoBERTa-base · GLUE QNLI fine-tuning | peak memory (GiB): 11.73 -> 10.77; 8.2% | Helped |
| Quantize activation backward gradients to 3 bits | RoBERTa-base · GLUE STS2 fine-tuning | peak memory (GiB): 13.31 -> 11.25; 15.5% | Helped |
| Replace vanilla GELU backward gradients with optimized 3-bit piecewise-constant gradients | RoBERTa-base · GLUE STSB, MRPC, RTE | mean accuracy/correlation: STSB: 0.909 ($\pm$ 0.001) -> 0.910 ($\pm$ 0.002); MRPC: 0.882 ($\pm$ 0.005) -> 0.884 ($\pm$ 0.007); RTE: 0.771 ($\pm$ 0. | Mixed |
| Use Few-bit backward instead of ActNN at 1-bit | RoBERTa-base · GLUE QQP | accuracy: $0.8880 \pm 0.0008$ -> $0.9080 \pm 0.0006$ | Helped |
| Use Few-bit backward instead of ActNN at 2-bit | RoBERTa-base · GLUE QQP | accuracy: $0.9072 \pm 0.0005$ -> $0.9097 \pm 0.0006$ | Helped |
| Use Few-bit backward instead of ActNN at 3-bit | RoBERTa-base · GLUE QQP | accuracy: $0.9106 \pm 0.0003$ -> $0.9114 \pm 0.0007$ | Helped |
| Use Few-bit backward instead of ActNN at 4-bit | RoBERTa-base · GLUE QQP | accuracy: $0.9113 \pm 0.0006$ -> $0.9112 \pm 0.0005$ | No clear effect |
Closest papers by meaning
- Don’t Waste Your Bits! Squeeze Activations and Gradients for Deep Neural Networks via TinyScript (ICML 2020 · similarity 0.90)
- Reducing Fine-Tuning Memory Overhead by Approximate and Memory-Sharing Backpropagation (ICML 2024 · similarity 0.88)
- Fine-tuning Quantized Neural Networks with Zeroth-order Optimization (ICLR 2026 · similarity 0.88)
- Network Memory Footprint Compression Through Jointly Learnable Codebooks and Mappings (ICLR 2024 · similarity 0.86)
- Better NTK Conditioning: A Free Lunch from (ReLU) Nonlinear Activation in Wide Neural Networks (NeurIPS 2025 · similarity 0.86)
- On the Duality between Gradient Transformations and Adapters (ICML 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.