Research map / Architectures and efficiency / Quantization
Learning from Loss Landscape: Generalizable Mixed-Precision Quantization via Adaptive Sharpness-Aware Gradient Aligning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes ASGA, a sharpness-aware gradient-aligning regularizer added to differentiable mixed-precision quantization so that bitwidth policies searched on small proxy datasets (e.g., CIFAR10) transfer to ImageNet and VOC with lower search cost.
Problem
Mixed-precision quantization (MPQ) policy search is computationally expensive on large datasets, and policies searched on small proxy datasets transfer poorly to large target datasets due to distribution shift and small data volume.
Approach
Minimize the sharpness (surrogate gap) of the quantized model's loss landscape during differentiable MPQ search, using an adaptive perturbation radius rho = min(rho_max, phi/ln(h+1)) and implicit gradient alignment (perturbed point at theta - mu*grad L) so the searched policy generalizes from small proxy data to large target data at SAM-level cost.
Key results
- On ResNet18 with 3-bit MPQ, ASGA on CIFAR10 proxy raised EdMIPS from 66.4 to 67.9 Top-1 on ImageNet while cutting convergence from 110 to 75 epochs.
- Using ASGA on CIFAR10 as proxy, the paper reports up to 150% search speedup (ResNet18), 127% (ResNet50), and 113% (MobileNet-V2) over SOTA MPQ methods.
- On VOC with ResNet18 Faster R-CNN, HAQ+ASGA improved mAP from 69.0 to 69.3 while reducing convergence epochs from 82 to 76.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add ASGA to EdMIPS | ResNet18 mixed-precision quantization · ImageNet | Top-1 accuracy: 66.4 -> 67.9 | Helped |
| Add ASGA to EdMIPS | ResNet18 mixed-precision quantization · ImageNet | Top-5 accuracy: 86.1 -> 87.7 | Helped |
| Add ASGA to HAQ | VGG16 with SSD · VOC | mAP: 69.4 -> 69.4 | No clear effect |
| Add ASGA to EdMIPS | VGG16 with SSD · VOC | mAP: 67.7 -> 67.8 | Helped |
| Add ASGA to HAQ | ResNet18 with Faster R-CNN · VOC | mAP: 69.0 -> 69.3 | Helped |
| Add ASGA to HAWQ | ResNet18 with Faster R-CNN · VOC | mAP: 66.4 -> 66.6 | Helped |
| Add ASGA to EdMIPS (ablation) | ResNet18 mixed-precision quantization · CIFAR10 proxy | Top-1 accuracy: 66.4 -> 67.9 | Helped |
| Add ASGA to EdMIPS (ablation) | ResNet18 mixed-precision quantization · ImageNet subset | Top-1 accuracy: 66.3 -> 67.3 | Helped |
| Add ASGA (ablation) | ResNet18 MPQ policy search · CIFAR10 proxy | average σ_max: 13.5 -> 11.4 | Helped |
| Add ASGA (ablation) | ResNet18 MPQ policy search · Flowers proxy | average σ_max: 48.4 -> 44.4 | Helped |
| Add ASGA (ablation) | ResNet18 MPQ policy search · Food proxy | average σ_max: 25.8 -> 23.1 | Helped |
| Use adaptive rather than fixed perturbation radius ρ (ablation) | ASGA policy search · CIFAR10 | convergence epochs | Helped |
Surprising findings
- Policies searched on the 60K-image ImageNet subset without ASGA underperform those searched on CIFAR10 with ASGA by about 1.6%, showing that direct search on target-distribution subsets is not sufficient.
- Directly applying HAQ and EdMIPS CIFAR10-searched policies to VOC causes significant performance degradation.
Limitations
- Speedup claims are measured in convergence epochs rather than wall-clock search time, and several reported gains are small (e.g., 0.1-0.3% Top-1 on several rows).
- Theoretical results rely on assumptions and a PAC-Bayes lemma whose proof is loosely argued; the rho adaptation is heuristic.
- Only CIFAR10, Flowers, and Food are tested as proxy datasets, and only ResNet/MobileNet/VGG architectures are evaluated.
Experimental setup
Datasets: CIFAR10 Flowers Food ImageNet VOC
Benchmarks: ImageNet VOC CIFAR10
Metrics: Top-1 accuracy Top-5 accuracy mAP search epochs / convergence epochs sharpness sigma_max
Baselines: ALQ EWGS HAWQ HAQ DQ HMQ GMPQ EdMIPS SEAM PACT
Builds on: SAM EdMIPS GMPQ EWGS ALQ
Closest papers by meaning
- InfoQ: Mixed-Precision Quantization via Global Information Flow (AAAI 2026 · similarity 0.91)
- Gradient Knows Best: Mixed-Precision Quantization via Gradient-Guided Bit Allocation for Super-Resolution (CVPR 2026 · similarity 0.90)
- Retraining-Free Model Quantization via One-Shot Weight-Coupling Learning (CVPR 2024 · similarity 0.89)
- Bit-Shrinking: Limiting Instantaneous Sharpness for Improving Post-Training Quantization (CVPR 2023 · similarity 0.89)
- Sharpness-Aware Data Generation for Zero-shot Quantization (ICML 2024 · similarity 0.87)
- MetaMix: Meta-State Precision Searcher for Mixed-Precision Activation Quantization (AAAI 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.