Research map / Architectures and efficiency / Quantization

Automatic Joint Structured Pruning and Quantization for Efficient Neural Network Training and Compression

CVPR 2025 · Architectures and efficiency › Quantization · approach: Post-training weight quantization · 21 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes GETA, a one-shot white-box framework that jointly applies structured pruning and quantization-aware training to any DNN via a quantization-aware dependency graph and a partially projected stochastic gradient optimizer (QASSO).

Problem

Joint structured pruning and quantization of DNNs is hard to use in practice due to multi-stage pipelines, black-box hyperparameter tuning for overall compression, and poor generalization beyond CNNs (e.g., transformers).

Approach

Builds a quantization-aware dependency graph to automatically define pruning search spaces for quantized networks, and uses a QASSO optimizer with partially projected SGD and quantization-aware forget-rate/step-size rules to explicitly control sparsity and per-layer bit widths in a single training run.

Key results

• On ResNet20/CIFAR10, GETA reaches 91.42% accuracy at 4.5% relative BOPs versus 91.70% baseline, vs ANNC at 90.90% and 6.1% BOPs.
• On BERT/SQuAD at 50% sparsity, GETA attains 76.74 EM / 85.87 F1 at 14.4% relative BOPs, outperforming OTO+8-bit PTQ (72.71 EM / 83.30 F1 at 16.7%).
• On ResNet50/ImageNet, GETA (50% sparsity) gets 74.40% accuracy at 5.38% relative BOPs versus Clip-Q's 73.70% at 6.30%.

Measured effects reported in the paper

Limitations

• Gradient-based full-gradient use limits scalability to larger LLMs (Phi2 chosen for feasibility on a single A100).
• On VGG7/CIFAR10 GETA's relative BOPs (0.41%) is worse than BB (0.29%).
• Only weight quantization is used in the ResNet20 comparison; LLM comparisons are against a prune-then-quantize baseline with uniform 8-bit PTQ.

Experimental setup

Datasets: CIFAR10 ImageNet2012 SQuAD LM-Evaluation-Harness commonsense benchmarks

Benchmarks: CIFAR10 ImageNet2012 SQuAD LM-Evaluation-Harness commonsense tasks (BoolQ, PIQA, HellaSwag, WinoGrande, ARC-e, ARC-c, OBQA)

Metrics: test accuracy EM F1 relative BOPs

Baselines: ANNC QST-B DJPQ BB Clip-Q OBC OTO followed by 8-bit PTQ SliceGPT LoraShear LoraPrune LLM-Pruner

Builds on: DepGraph OTO HESSO parameterized quantization layer (QAT with learnable step size)

Closest papers by meaning

• Automatic Neural Network Compression by Sparsity-Quantization Joint Learning: A Constrained Optimization-Based Approach (CVPR 2020 · similarity 0.90)
• Harmonious Coexistence of Structured Weight Pruning and Ternarization for Deep Neural Networks (AAAI 2020 · similarity 0.89)
• CLIP-Q: Deep Network Compression Learning by In-Parallel Pruning-Quantization (CVPR 2018 · similarity 0.89)
• Pruning vs Quantization: Which is Better? (NeurIPS 2023 · similarity 0.89)
• Learning Pruning-Friendly Networks via Frank-Wolfe: One-Shot, Any-Sparsity, And No Retraining (ICLR 2022 · similarity 0.89)
• BMRS: Bayesian Model Reduction for Structured Pruning (NeurIPS 2024 · similarity 0.88)