Research map / Architectures and efficiency / Pruning and efficient tuning
Preserving Deep Representations in One-Shot Pruning: A Hessian-Free Second-Order Optimization Framework
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes SNOWS, a one-shot post-training pruning method that optimizes a K-step nonlinear layer-wise reconstruction loss using Hessian-free Newton descent with CG, applied on top of existing masks for CNNs and ViTs.
Problem
One-shot post-training pruning of vision networks without retraining; existing layer-wise least-squares reconstruction ignores nonlinear effects on deeper representations, while global second-order methods are intractable for large networks.
Approach
Replace the layer-wise linear reconstruction loss with a K-step objective that reconstructs dense outputs of subsequent nonlinear operations, and solve it with a mask-agnostic, exact Hessian-vector-product Newton/CG method (Hessian-free optimization) applied layer by layer with cascaded inputs.
Key results
- ResNet50 at 1:4 sparsity on CIFAR-100 drops 1.5% accuracy with SNOWS vs 7.6% for prior best one-shot N:M method; on ImageNet-1k drops 9.4% vs 29.42%.
- MobileNetV1 at 70% unstructured sparsity on ImageNet-1k loses 5.61% (65.28%) with SNOWS vs 12.55% for CHITA.
- ViT-B/16 at 2:4 sparsity (QKV, out, MLP pruned) reaches 76.57% top-1 vs 70.89% for MP, from 80.42% dense.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Apply SNOWS weight optimization to an MP mask instead of using MP alone | ViT-B/16 · ImageNet-1k, 2:4 sparsity, QKV | Top-1 accuracy: 78.97 -> 79.45 | Helped |
| Apply SNOWS weight optimization to an MP mask instead of using MP alone | ViT-B/16 · ImageNet-1k, 2:4 sparsity, QKV, output projection, MLP | Top-1 accuracy: 70.89 -> 76.57 | Helped |
| Apply SNOWS weight optimization to an MP mask instead of using MP alone | ViT-L/16 · ImageNet-1k, 2:4 sparsity, QKV | Top-1 accuracy: 75.55 -> 81.01 | Helped |
| Apply SNOWS weight optimization to an MP mask instead of using MP alone | ViT-L/16 · ImageNet-1k, 2:4 sparsity, MLP | Top-1 accuracy: 0.28 -> 69.71 | Helped |
| Apply SNOWS from a magnitude-pruning mask instead of using FALCON | ResNet20 · CIFAR-10, unstructured sparsity levels 0.5–0.9 | accuracy: 90.87 -> 92.28; 89.67 -> 91.90; 84.42 -> 90.87; 65.17 -> 87.82; 19.14 -> 64.80 | Helped |
| Apply SNOWS from a magnitude-pruning mask instead of using FALCON | MobileNetV1 · ImageNet-1K, unstructured sparsity levels 0.5–0.8 | accuracy: 70.35 -> 70.10; 67.18 -> 68.70; 58.40 -> 65.28; 25.82 -> 54.83 | Mixed |
| Increase the reconstruction horizon K (ablation) | SNOWS · ResNet20 CIFAR-10 and ResNet50 CIFAR-100 | accuracy | Helped |
| Replace Newton optimization with SGD for the K-step reconstruction objective (ablation) | SNOWS optimization · ResNet20 CIFAR-10 and ResNet50 ImageNet-1k, first layer, 2:4 mask | reconstruction loss | Hurt |
| Replace the exact Hessian with the Fisher approximation (ablation) | SNOWS optimization · ResNet20 and ResNet50 reconstruction | convergence speed | Hurt |
| Lower the maximum number of CG iterations (ablation) | SNOWS optimization · Reconstruction across tested layers and models | reconstruction loss | Mixed |
| Fine-tune the pruned SNOWS model while maintaining its pruning mask (ablation) | SNOWS · ResNet20 CIFAR-10 and MobileNetV1 ImageNet-1K, unstructured sparsity | accuracy: 92.28 -> 92.32; 91.90 -> 92.13; 90.87 -> 91.85; 87.82 -> 91.06; 64.80 -> 87.69; 70.10 -> 70.66; 68.70 -> 70.10; 65.28 -> | Helped |
Surprising findings
- SGD converges very slowly on the sparse reconstruction problem (about 2000 steps and still far from solution), while Newton-CG converges in fewer than 10 steps.
- Using the exact Hessian-vector product rather than a Fisher approximation yields faster convergence in both iterations and runtime.
Limitations
- SNOWS does not perform mask selection and relies on an external mask (e.g. magnitude pruning).
- Runtime and memory grow with K, causing out-of-memory for large K on ResNet50 ImageNet.
- N:M speed-ups are reported as FLOPs reduction rather than measured latency due to limited kernel support.
Experimental setup
Datasets: CIFAR-10 CIFAR-100 ImageNet-1k MiniImageNet-1k
Benchmarks: CIFAR-10 CIFAR-100 ImageNet-1k MiniImageNet-1k
Metrics: top-1 accuracy top-5 accuracy accuracy change vs dense baseline run-time peak GPU memory
Baselines: Magnitude Pruning OBC SparseGPT WoodFisher Combinatorial Brain Surgeon CHITA FALCON
Builds on: Optimal Brain Surgeon Optimal Brain Compression Net-Trim SparseGPT Hessian-free optimization
Closest papers by meaning
- Learning Pruning-Friendly Networks via Frank-Wolfe: One-Shot, Any-Sparsity, And No Retraining (ICLR 2022 · similarity 0.90)
- OSSCAR: One-Shot Structured Pruning in Vision and Language Models with Combinatorial Optimization (ICML 2024 · similarity 0.89)
- Learnable Sparsity for Vision Generative Models (ICLR 2026 · similarity 0.88)
- Elastic ViTs from Pretrained Models without Retraining (NeurIPS 2025 · similarity 0.88)
- UPDP: A Unified Progressive Depth Pruner for CNN and Vision Transformer (AAAI 2024 · similarity 0.87)
- Adaptive Sharpness-Aware Pruning for Robust Sparse Networks (ICLR 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.