Research map / Architectures and efficiency / Pruning and efficient tuning
Towards Meta-Pruning via Optimal Transport
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Intra-Fusion, a meta-pruning method that uses optimal transport to fuse pruned neurons into surviving ones instead of discarding them, recovering accuracy without fine-tuning or data, and factorizes training via split-data fusion.
Problem
Structured pruning of neural networks usually discards less important neurons, causing large accuracy drops that require costly fine-tuning; the pruning procedure itself, not only the importance metric, is treated as fixed.
Approach
Instead of removing low-importance neuron pairings, use optimal transport (as in OTFusion) to compute a transport map that merges all neurons of a group into the target count, so discarded neurons' information is aggregated into the kept ones; works with any agnostic importance metric and needs no data.
Key results
- Data-free ResNet50 on ImageNet with Taylor importance: Intra-Fusion gains up to about +60% accuracy over conventional pruning at high sparsity in some groups (e.g., Group 6 at 40% neuron sparsity: 80.2% to 92.1%).
- With fine-tuning on CIFAR-10, Intra-Fusion gives consistent gains of +0.24 to +0.96 points over default l1 pruning for VGG11-BN and ResNet18 across 64-96% weight sparsity.
- Split-data PaF and FaP give overall speedups of 1.42 and 1.31 for VGG11-BN on CIFAR-10 relative to whole-data training.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Replace default pruning with Intra-Fusion | VGG11-BN · CIFAR-10, 64%–96% sparsity, fine-tuned | accuracy: +0.24; +0.36; +0.72; +0.68 | Helped |
| Replace default pruning with Intra-Fusion | ResNet18 · CIFAR-10, 64%–96% sparsity, fine-tuned | accuracy: +0.45; +0.40; +0.96; +0.79 | Helped |
| Use random importance scores with Intra-Fusion instead of random pruning (ablation) | ResNet50 · ImageNet, Group 0, 40% sparsity | accuracy: 53.94 -> 70.18; +16.24 | Helped |
| Use importance-informed source and target distributions rather than uniform distributions (ablation) | Intra-Fusion · CIFAR-10 and CIFAR-100, ResNet18 | accuracy | No clear effect |
| Use importance-informed source distribution with uniform target rather than uniform source and target (ablation) | Intra-Fusion · CIFAR-100, ResNet18, Groups 10 and 11 | accuracy: 94.84 -> 94.49 | Hurt |
| Split training data and train models in parallel before fusion | VGG11-BN · CIFAR-10 | training speedup: 1.81 | Helped |
| Extend the k-fold split from 2-Fold to 4-Fold (ablation) | VGG11-BN · Split-data training | accuracy: up to 1% | Helped |
| Extend the k-fold split from 2-Fold to 4-Fold (ablation) | ResNet18 · Split-data training | accuracy | No clear effect |
Surprising findings
- Intra-Fusion improves accuracy even when neuron importance is random, suggesting the gain comes from merging discarded neurons rather than from the importance metric.
- Importance-informed versus uniform OT mass distributions give no statistically significant difference in most group-sparsity cases.
Limitations
- Gains after fine-tuning are modest (around 1 point) and the paper concedes the focus is data-free pruning.
- Split-data speedups are shown only for small CIFAR-10 setups; larger-scale gains are speculated rather than measured.
- Importance-informed distributions show no clear advantage with the l1 criterion, leaving the role of the importance metric unclear.
Experimental setup
Datasets: CIFAR-10 CIFAR-100 ImageNet
Benchmarks: CIFAR-10 CIFAR-100 ImageNet
Metrics: test accuracy output l2 divergence training speedup
Baselines: l1-norm pruning Taylor importance pruning LAMP CHIP default pruning with fine-tuning
Builds on: OTFusion DepGraph structural pruning Lottery Ticket Hypothesis
Closest papers by meaning
- Importance Estimation for Neural Network Pruning (CVPR 2019 · similarity 0.90)
- Pruning neural networks without any data by iteratively conserving synaptic flow (NeurIPS 2020 · similarity 0.89)
- Pruning via Sparsity-indexed ODE: a Continuous Sparsity Viewpoint (ICML 2023 · similarity 0.89)
- “Learning-Compression” Algorithms for Neural Net Pruning (CVPR 2018 · similarity 0.89)
- Neuron Merging: Compensating for Pruned Neurons (NeurIPS 2020 · similarity 0.89)
- A Layer Decomposition-Recomposition Framework for Neuron Pruning towards Accurate Lightweight Networks (AAAI 2019 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.