Research map / Optimization / Optimizers
One-Step Forward and Backtrack: Overcoming Zig-Zagging in Loss-Aware Quantization Training
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes BLAQ, a one-step forward-and-backtrack loss-aware weight quantization method that uses a trial next-step gradient to damp zig-zagging oscillations and speed convergence in 1-bit and low-bit weight quantization.
Problem
Loss-aware weight quantization suffers from a zig-zagging-like oscillation of gradient directions caused by quantized-gradient errors, which slows convergence and can prevent it, especially at extremely low bitwidths.
Approach
A one-step forward search computes a trial quantized gradient at the next step; the current gradient and moments are then blended with this trial gradient (backtracking) to update full-precision and quantized weights, giving a more accurate and stable gradient direction than standard LAQ.
Key results
- On ImageNet with ResNet18, BLAQ reaches 66.73% Top-1 at 1-bit and 69.62% at 2-bit, versus 69.8% full precision.
- On CIFAR10 with VGG, BLAQ 1-bit reaches 91.5% Top-1 versus 89.5% for LAQ and 92.8% full precision.
- On MNIST with LeNet5, BLAQ 1-bit reaches 99.38% Top-1, above full precision at 99.19%.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use one-step forward search and backtracking updates instead of LAQ updates | BLAQ on VGG · CIFAR10, 1-bit | Top-1 accuracy: 91.2% -> 91.5% | Helped |
| Use one-step forward search and backtracking updates instead of LAQ updates | BLAQ on 4-layer Model · MNIST, 1-bit | Top-1 accuracy: 98.82% -> 99.11% | Helped |
| Use one-step forward search and backtracking updates instead of the strongest listed baseline | BLAQ on LeNet · MNIST, 1-bit | Top-1 accuracy: 99.26% -> 99.38% | Helped |
| Use one-step forward search and backtracking updates instead of LAQ updates | BLAQ on SVHNNet · SVHN, 1-bit | Top-1 accuracy: 97.64% -> 98.13% | Helped |
| Use one-step forward search and backtracking updates instead of the strongest listed baseline | BLAQ on ResNet18 · ILSVRC12, 1-bit | Top-1 accuracy: 65.6% -> 66.73% | Helped |
| Use one-step forward search and backtracking updates instead of the strongest listed baseline | BLAQ on ResNet18 · ILSVRC12, 2-bit | Top-1 accuracy: 68.9% -> 69.62% | Helped |
| Replace LAQ with BLAQ updates | ResNet18 quantization · ILSVRC12, 1-bit | maximum oscillation amplitude: 0.000486 -> 0.000143 | Helped |
| Replace LAQ with BLAQ updates | ResNet18 quantization · ILSVRC12, 1-bit | oscillation frequency: 15 -> 4 | Helped |
| Replace LAQ with BLAQ updates | ResNet18 quantization · ILSVRC12, 2-bit | maximum oscillation amplitude: 0.000342 -> 0.000102 | Helped |
| Replace LAQ with BLAQ updates | ResNet18 quantization · ILSVRC12, 2-bit | oscillation frequency: 11 -> 3 | Helped |
| Replace LAQ with BLAQ updates | ResNet18 quantization · ILSVRC12, 4-bit | maximum oscillation amplitude: 0.000261 -> 0.000067 | Helped |
| Replace LAQ with BLAQ updates | ResNet18 quantization · ILSVRC12, 4-bit | oscillation frequency: 8 -> 1 | Helped |
Surprising findings
- BLAQ 1-bit and 2-bit weights outperform the full-precision baselines on MNIST and SVHN, which the authors attribute to reduced redundancy.
Limitations
- Hyperparameters a and m are tuned per dataset class and the appendix link is the only supporting material.
- Convergence claims rely on convex, smooth, strongly convex assumptions that do not hold for deep networks.
- LAQ baseline numbers are taken from the original authors, and LAQ has no ImageNet results.
Experimental setup
Datasets: CIFAR10 MNIST SVHN ILSVRC12 (ImageNet)
Benchmarks: CIFAR10 MNIST SVHN ILSVRC12 (ImageNet)
Metrics: Top-1 accuracy training loss convergence speed
Baselines: BC BWN LAQ TRQ AQ ALQ DC ADMM LR DSQ TWN LQ-Net
Builds on: loss-aware quantization (LAQ) BinaryConnect proximal Newton method
Closest papers by meaning
- Loss-aware Weight Quantization of Deep Networks (ICLR 2018 · similarity 0.89)
- Analysis of Quantized Models (ICLR 2019 · similarity 0.89)
- Gradient $\ell_1$ Regularization for Quantization Robustness (ICLR 2020 · similarity 0.89)
- Explicit Loss-Error-Aware Quantization for Low-Bit Deep Neural Networks (CVPR 2018 · similarity 0.89)
- Zero-Shot Quantization via Weight-Space Arithmetic (NeurIPS 2026 · similarity 0.86)
- Retraining-Free Model Quantization via One-Shot Weight-Coupling Learning (CVPR 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.