AI Paper Atlas

Research map / Learning theory and algorithms / Prediction and decision losses

WoodFisher: Efficient Second-Order Approximation for Neural Network Compression

Sidak Pal Singh, Dan Alistarh

NeurIPS 2020 · Learning theory and algorithms › Prediction and decision losses · approach: Theory of neural network sensitivity · 36 citations (OpenAlex)

Read the paperCheck your idea against 137,217 papers

AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Measured effects reported in the paper

What was changedWhereMeasuredEffect
Use joint/global second-order pruning instead of global magnitude pruningWoodFisher pruning · ImageNet, ResNet-50, 98% sparsityTop-1 accuracy: 64.28 -> 65.55Improved
Use WoodFisher instead of STRResNet-50 gradual pruning · ImageNet, 80% sparsityTop-1 accuracy: 76.19 -> 76.76Improved
Use WoodFisher instead of global magnitude pruningMobileNetV1 gradual pruning · ImageNet, 89.00% sparsityTop-1 accuracy: 63.02 -> 63.87Improved
Increase WoodFisher diagonal-block chunk size (ablation)Joint WoodFisher · CIFAR10, ResNet-20 one-shot pruningtest accuracyImproved
Use mini-batch-averaged gradients instead of individual-example gradients for Fisher (ablation)WoodFisher · ImageNet, ResNet-50 one-shot pruningTop-1 accuracyImproved
Increase Fisher mini-batch size from 1 to 100 (ablation)WoodFisher · ImageNet, ResNet-50 one-shot pruning at 30% sparsityTop-1 accuracy: 55.81 ± 3.28 -> 75.73 ± 0.03Improved
Use 160 rather than 80 Fisher subsamples (ablation)WoodFisher · ImageNet, ResNet-50 one-shot pruning at 30% sparsity, mini-batch size 100Top-1 accuracy: 75.73 ± 0.03 -> 75.77 ± 0.08No clear effect
Recompute the inverse-Hessian estimate during pruning without retraining (ablation)WoodFisher · ImageNet, ResNet-50 one-shot pruningTop-1 accuracyImproved
Use model-sampled labels instead of empirical labels to compute Fisher gradients (ablation)WoodFisher · ImageNet, ResNet-50 one-shot pruningTop-1 accuracy: 76.10 ± 0.04 -> 76.16 ± 0.02; 75.22 ± 0.07 -> 75.31 ± 0.05; 69.21 ± 0.05 -> 69.29 ±0.20; 48.35 ± 0.22 -> 48.74 ±1.03; 5.No clear effect
Include the first-order gradient term in pruning importance (WoodTaylor) (ablation)WoodFisher pruning · MNIST, partially trained MLPNetaccuracy: about 5%Improved
Include the first-order gradient term in pruning importance (WoodTaylor) (ablation)WoodFisher pruning · CIFAR10, ResNet-20accuracyImproved
Replace WoodFisher with diagonal-Fisher pruning (ablation)One-shot pruning · CIFAR10, ResNet-20 above 30% sparsitytest accuracyHurt

Closest papers by meaning

Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.

Open scime draft

Get the next scime Dispatch and a monthly field update

Free. We send a confirmation link first; every email has a one-click unsubscribe. Privacy