Research map / Architectures and efficiency / Pruning and efficient tuning
Optimal Brain Apoptosis
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Optimal Brain Apoptosis (OBA), a Hessian-based pruning method that computes exact per-parameter Hessian-vector product terms for CNNs and Transformers via series/parallel connectivity analysis and a forward-mode JVP trick, evaluated for structured and unstructured pruning.
Problem
Parameter importance estimation for pruning CNNs and Transformers relies on Hessian approximations (diagonal, Fisher) that are inaccurate for capturing second-order loss changes from removing parameters.
Approach
Decomposes the network Hessian into inter-layer submatrices, identifies when they are nonzero (series and parallel connectivity, e.g., matrix multiplication in attention), and computes the exact Hessian-vector product per parameter using a Jacobian-vector-product forward propagation, avoiding Fisher approximations.
Key results
- On ImageNet ResNet50, OBA reaches 75.62% (-0.53) at 2.00x speedup, versus 75.48% for C-OBS and 75.12% for weight pruning.
- On ImageNet ViT-B/16, OBA achieves 79.64% (-1.43) at 1.30x speedup, versus 77.03% for weight pruning and 77.65% for Taylor.
- On unstructured ResNet-20 CIFAR-10 at 90% sparsity, OBA reaches 86.27% accuracy versus 67.00% for weight pruning and about 10% for Taylor and OBD.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use OBA instead of C-OBS for structured pruning | ResNet50 · ImageNet | accuracy (%): 75.48 -> 75.62 | Helped |
| Use OBA instead of Taylor for structured pruning | ViT-B/16 · ImageNet | accuracy (%): 77.65 -> 79.64 | Helped |
| Use OBA instead of CHITA++ for unstructured pruning | ResNet-20 · CIFAR-10, sparsity 0.9 | accuracy (%): 79.32 -> 86.27 | Helped |
| Use OBA instead of CHITA-BSO++ for unstructured pruning | ResNet-50 · ImageNet, sparsity 0.8 | accuracy (%): 26.21 -> 47.14 | Helped |
| Use OBA instead of CHITA-BSO++ for unstructured pruning | ResNet-50 · ImageNet, sparsity 0.9 | accuracy (%): 0.43 -> 5.65 | Helped |
Surprising findings
- Taylor and OBD collapse to about 10% accuracy in the first unstructured pruning step on ResNet-20/ResNet-50, while weight magnitude pruning remains usable.
Limitations
- Applicability limited to MLPs, CNNs and Transformers; RNNs and state space models not covered.
- Computing the parallel-connectivity terms adds substantial overhead (about 4.0 s/iteration vs 0.3 s regular training on ResNet32 vs ViT).
Experimental setup
Datasets: CIFAR10 CIFAR100 ImageNet
Benchmarks: CIFAR10 CIFAR100 ImageNet
Metrics: accuracy FLOPs reduction weight reduction Spearman correlation running time
Baselines: Weight Taylor OBD OBS C-OBD C-OBS Kron-OBD Kron-OBS EigenDamage CHITA NN Slimming
Builds on: Optimal Brain Damage Optimal Brain Surgeon EigenDamage CHITA
Closest papers by meaning
- Importance Estimation for Neural Network Pruning (CVPR 2019 · similarity 0.86)
- The Iterative Optimal Brain Surgeon: Faster Sparse Recovery by Leveraging Second-Order Information (NeurIPS 2024 · similarity 0.85)
- Towards Meta-Pruning via Optimal Transport (ICLR 2024 · similarity 0.84)
- Optimal Eye Surgeon: Finding image priors through sparse generators at initialization (ICML 2024 · similarity 0.84)
- How Sparse Can We Prune A Deep Network: A Fundamental Limit Perspective (NeurIPS 2024 · similarity 0.83)
- Preserving Deep Representations in One-Shot Pruning: A Hessian-Free Second-Order Optimization Framework (ICLR 2025 · similarity 0.83)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.