Research map / Architectures and efficiency / Pruning and efficient tuning
Compression-aware Training of Deep Networks
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's title only (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Train with a low-rank regularizer and apply low-rank post-processing at e_l=80% | Dec8^512 · ImageNet | top-1 accuracy: 67.0 -> 66.8 | No clear effect |
| Train with a low-rank regularizer and apply low-rank post-processing at e_l=100% | Dec8^512 · ImageNet | top-1 accuracy: 67.0 -> 67.6 | Helped |
| Train with low-rank regularization and group sparsity | Dec3^512 · ICDAR | top-1 accuracy: 88.8 -> 89.7 | Helped |
| Use low-rank regularization with group sparsity and retain all energy | Dec3^512 · ICDAR | top-1 accuracy: 89.3 -> 90.5 | Helped |
| Train with a low-rank regularizer and use SVD post-processing at e_l=80% | Dec3^512 · ICDAR | parameter reduction: -91.9 | Helped |
| Train with low-rank regularization, then add group sparsity | ResNet-50 · ImageNet | compression rate: 20.6% -> 27% | Helped |
| Apply low-rank post-processing to a conventionally trained Dec8^512 model instead of training compression-aware | Dec8^512 · ImageNet | parameter reduction: -39.5 -> -53.5 | Hurt |
| Use e_l=80% instead of e_l=100% in low-rank post-processing (ablation) | Dec8^512 · ImageNet | top-1 accuracy: 67.6 -> 66.8 | Mixed |
| Prune and reload the model after 15 epochs, using low-rank regularization and group sparsity | Compression-aware model · Training experiment | training time: 1.69 to 0.77 hours | Helped |
| Prune and reload the model after 15 epochs, using low-rank regularization and group sparsity | Compression-aware model · Training experiment | number of parameters: 3.7M to 210K | Helped |
| Use e_l=80% instead of e_l=100% for Dec8^256 low-rank post-processing (ablation) | Dec8^256-τ=5 · Titan X (Pascal) inference | forward time: 119.27 -> 88.33 | Helped |
| Use SVD post-processing rather than leaving the low-rank-regularized network undecomposed (ablation) | Dec8^256-τ=5 · Titan X (Pascal) inference | forward time: 90.55 -> 88.33 | No clear effect |
Closest papers by meaning
- Model Preserving Compression for Neural Networks (NeurIPS 2022 · similarity 0.91)
- Compressing Neural Networks: Towards Determining the Optimal Layer-wise Decomposition (NeurIPS 2021 · similarity 0.90)
- "Lossless" Compression of Deep Neural Networks: A High-dimensional Neural Tangent Kernel Approach (NeurIPS 2022 · similarity 0.90)
- Distribution-Aware Tensor Decomposition for Compression of Convolutional Neural Networks (NeurIPS 2025 · similarity 0.89)
- Auto-Compressing Networks (NeurIPS 2025 · similarity 0.88)
- Dynamical Low-Rank Compression of Neural Networks with Robustness under Adversarial Attacks (NeurIPS 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.