Research map / Architectures and efficiency / Pruning and efficient tuning
Compressing Models With Few Samples: Mimicking Then Replacing
Huanyu Wang, Junjie Liu, Xin Ma, Yang Yong, Zhenhua Chai, Jianxin Wu
Check your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Mimic features before rather than after final pooling (ablation) | MiR on ResNet-34 · ILSVRC-2012, Prune-C Normal, 500 samples | top-1 accuracy: 65.4 -> 67.0 | Improved |
| Mimic features before rather than after final pooling (ablation) | MiR on ResNet-34 · ILSVRC-2012, Prune-C Residual, 500 samples | top-1 accuracy: 62.4 -> 65.4 | Improved |
| Use holistic feature matching instead of layer-wise reconstruction | Pruned ResNet-34 · ILSVRC-2012, Prune-C Normal, 500 samples | top-1 accuracy: CD 63.7 -> MiR_before 67.0 | Improved |
| Use holistic feature matching instead of layer-wise reconstruction | Pruned ResNet-34 · ILSVRC-2012, Prune-C Residual, 500 samples | top-1 accuracy: FSKD 42.8 -> MiR_before 65.4 | Improved |
| Use MiR feature mimicking on MobileNetV2 rather than label-based fine-tuning | Pruned MobileNetV2 · ILSVRC-2012, Prune-D, 500 samples | top-1 accuracy: KD 48.4 -> MiR_before 67.6 | Improved |
| Use MiR_before rather than CD | Pruned ResNet-56 · CIFAR-10, 1 sample per class | top-1 accuracy: 89.00 -> 89.27 | Improved |
| Replace MSE with L1 feature loss (ablation) | MiR on ResNet-34 · ILSVRC-2012, Prune-B CD-style, 500 samples | top-1 accuracy: 69.75 -> 69.88 | No clear effect |
| Replace MSE with cosine similarity loss (ablation) | MiR on ResNet-34 · ILSVRC-2012, Prune-B CD-style, 500 samples | top-1 accuracy: 69.75 -> 69.07 | Hurt |
| Replace MSE with LSH loss (ablation) | MiR on ResNet-34 · ILSVRC-2012, Prune-B CD-style, 500 samples | top-1 accuracy: 69.75 -> 66.89 | Hurt |
| Add L1 loss to MSE (ablation) | MiR on ResNet-34 · ILSVRC-2012, Prune-B CD-style, 500 samples | top-1 accuracy: 69.75 -> 69.89 | No clear effect |
| Tune the classifier after mimicking, while freezing the trained backbone (ablation) | MiR · ILSVRC-2012, Prune-B CD-style, 500 samples | top-1 accuracy: 69.75 -> 66.44 (LR 0.1); 69.75 -> 69.96 (LR 0.05) | Mixed |
| Tune the classifier while freezing an untrained backbone (ablation) | Pruned ResNet-34 · ILSVRC-2012, Prune-B CD-style, 500 samples | top-1 accuracy: 69.75 -> 45.43 (LR 0.1) | Hurt |
Earlier methods this paper compares with
Named baselines from earlier accepted papers in this paper's results tables: better than 1 of 1 on most shared results.
- ThiNet — ThiNet: A Filter Level Pruning Method for Deep Neural Network Compression (ICCV 2017) · better on 1 of 1 shared result
Matched by method name in results tables read by AI. Authors' own tables tend to favour the newer method, and a comparison can be missed when a method is renamed.
Closest papers by meaning
- MCNC: Manifold-Constrained Reparameterization for Neural Compression (ICLR 2025 · similarity 0.85)
- 3BASiL: An Algorithmic Framework for Sparse plus Low-Rank Compression of LLMs (NeurIPS 2025 · similarity 0.85)
- Forget the Data and Fine-Tuning! Just Fold the Network to Compress (ICLR 2025 · similarity 0.85)
- Compressing Large Language Models using Low Rank and Low Precision Decomposition (NeurIPS 2024 · similarity 0.85)
- EPSD: Early Pruning with Self-Distillation for Efficient Model Compression (AAAI 2024 · similarity 0.84)
- Streamlining Redundant Layers to Compress Large Language Models (ICLR 2025 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
