Research map / Architectures and efficiency / Pruning and efficient tuning
MaxQ: Multi-Axis Query for N:M Sparsity Network
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes MaxQ, a parameter-free multi-axis query method that generates soft N:M sparsity masks from weight magnitudes across filter and kernel axes, combined with an incremental N:M block schedule, to train CNNs with N:M sparsity from scratch.
Problem
N:M sparsity methods treat weights within each block independently, ignoring relative importance across blocks, and apply N:M sparsity to the whole network from the start of training, causing severe information loss and suboptimal accuracy.
Approach
Relax hard N:M binary masks into soft masks computed via sigmoid thresholds over weight magnitudes, with importance queried along filter and kernel axes, and gradually increase the fraction of N:M blocks during training (largest l1-norm blocks first) so the network heals progressively; the soft masks are precomputed and folded into weights at inference.
Key results
- ResNet50 with 1:16 sparsity reaches 74.6% top-1 on ImageNet, 2.8% above the prior best (LBC/SR-STE-based methods at 71.8%/71.5%).
- ResNet50 with 2:4 sparsity reaches 77.6% top-1, above the dense baseline of 77.3%.
- On COCO with Faster R-CNN at 2:4 sparsity, MaxQ achieves 38.7 mAP vs. 37.4 dense baseline.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use multi-axis query with soft masks (ablation) | MaxQ on ResNet50 · ImageNet, 1:16 sparsity | Top-1 accuracy: 71.5% -> 74.2% | Helped |
| Add incremental pruning to multi-axis query (ablation) | MaxQ on ResNet50 · ImageNet, 1:16 sparsity | Top-1 accuracy: 74.2% -> 74.6% | Helped |
| Train for 200 epochs instead of the default schedule (ablation) | MaxQ on ResNet50 · ImageNet, 1:16 sparsity | Top-1 accuracy: 74.6% -> 75.2% | Helped |
| Use multi-axis query and incremental pruning (ablation) | ResNet50 · ImageNet, 1:16 sparsity | Top-1 accuracy: 71.5% -> 74.6% | Helped |
| Use MaxQ instead of ASP | ResNet50 · ImageNet, 1:16 sparsity | Top-1 accuracy: 71.5% -> 74.6% | Helped |
| Use MaxQ instead of ASP | ResNet50 · ImageNet, 1:4 sparsity | Top-1 accuracy: 76.5% -> 77.3% | Helped |
| Use MaxQ instead of LBC | Faster R-CNN · COCO, 2:4 sparsity | mAP: 38.5 -> 38.7 | Helped |
| Use MaxQ instead of LBC | Mask R-CNN · COCO, 2:4 sparsity | Box mAP: 39.3 -> 39.2 | Hurt |
| Use MaxQ instead of LBC | Mask R-CNN · COCO, 1:4 sparsity | Box mAP: 37.8 -> 38.3 | Helped |
| Use MaxQ instead of LBC | Mask R-CNN · COCO, 1:4 sparsity | Mask mAP: 34.0 -> 34.4 | Helped |
| Use varied incremental-pruning start and end epochs (ablation) | MaxQ on ResNet50 · ImageNet, 1:16 sparsity | Top-1 accuracy: 74.51%, 74.41%, 74.55%, 74.56%, 74.52%, 74.47% | No clear effect |
Surprising findings
- MaxQ at 2:4 sparsity matches or exceeds the dense ResNet34, ResNet50 and MobileNetV1 baselines.
- Applying N:M sparsity first to blocks with larger l1-norm gives more stable convergence than the reverse order, and the PPF pipeline does not significantly help N:M networks.
Limitations
- Training speed is about 15% slower than SR-STE due to memory access in the multi-axis query.
- Evaluated only on CNN architectures and vision tasks.
Experimental setup
Datasets: ImageNet COCO
Benchmarks: ImageNet COCO
Metrics: top-1 accuracy mAP FLOPs train speed INT8 accuracy drop
Baselines: ASP SR-STE LBC RigL GMP MAP STR DNW OptG
Builds on: SR-STE ASP LBC
Closest papers by meaning
- Spartan: Differentiable Sparsity via Regularized Transportation (NeurIPS 2022 · similarity 0.87)
- Learning Best Combination for Efficient N:M Sparsity (NeurIPS 2022 · similarity 0.87)
- Sparsity May Cry: Let Us Fail (Current) Sparse Neural Networks Together! (ICLR 2023 · similarity 0.87)
- From Hard to Soft: Understanding Deep Network Nonlinearities via Vector Quantization and Statistical Inference (ICLR 2019 · similarity 0.86)
- MultiMax: Sparse and Multi-Modal Attention Learning (ICML 2024 · similarity 0.86)
- Fast and Controllable Post-training Sparsity: Learning Optimal Sparsity Allocation with Global Constraint in Minutes (AAAI 2024 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.