Research map / Architectures and efficiency / Pruning and efficient tuning
Distribution-Aware Tensor Decomposition for Compression of Convolutional Neural Networks
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes data-informed (covariance-weighted, functional) norm for tensor decomposition of CNN conv layers, with new ALS algorithms for Tucker-2 and CP that minimize the output-distribution change, achieving competitive accuracy without fine-tuning.
Problem
Low-rank tensor compression of CNNs minimizes isotropic weight-space (Frobenius) error, which ignores the effect on layer outputs and typically requires costly fine-tuning that needs the original training data.
Approach
Replace the Frobenius weight-space error with a functional norm ||(W - W~) Sigma^{1/2}||_F, where Sigma is the input covariance of the layer (unfolded patches), and derive closed-form ALS updates for Tucker-2 and CP under this norm; the covariance can be estimated from a different dataset than the training set.
Key results
- On ResNet18 ImageNet with CP, CP-ALS-Sigma reaches 67.9% top-1 at compression 2.07 versus 35.2% for greedy tensor deflation under the same norm.
- On GoogLeNet with Tucker2 at compression 1.51, the proposed method attains 52.2% vs 23.6% for Tucker2-ALS (Frobenius).
- On ResNet18 (CIFAR-10, CP-ALS-Sigma) accuracy drops only 0.8% at compression 3.53 relative to original 94.3%, without fine-tuning.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use Tucker2-ALS-Sigma instead of Tucker2-ALS | GoogLeNet · GoogLeNet compression rates 1.24, 1.33, 1.42, and 1.51 | accuracy: 60.2 vs 66.3; 52.5 vs 64.2; 39.9 vs 60.1; 23.6 vs 52.2 | Helped |
| Use Tucker2-ALS-Sigma instead of AG-LRA | GoogLeNet · GoogLeNet compression rates 1.24, 1.33, 1.42, and 1.51 | accuracy: 60.5 vs 66.3; 54.9 vs 64.2; 43.0 vs 60.1; 25.1 vs 52.2 | Helped |
| Use Tucker2-ALS-Sigma instead of FW-LRA | GoogLeNet · GoogLeNet compression rates 1.24, 1.33, 1.42, and 1.51 | accuracy: 53.0 vs 66.3; 35.8 vs 64.2; 6.0 vs 60.1; 1.1 vs 52.2 | Helped |
| Use Tucker2-ALS-Sigma instead of Tucker2-ALS | ResNet18 · ResNet18 compression rates 1.4, 1.5, 1.63, and 1.77 | accuracy: 58.9 vs 66.8; 52.2 vs 66.1; 36.1 vs 64.9; 30.2 vs 63.3 | Helped |
| Use Tucker2-ALS-Sigma instead of AG-LRA | ResNet18 · ResNet18 compression rates 1.4, 1.5, 1.63, and 1.77 | accuracy: 54.9 vs 66.8; 45.0 vs 66.1; 37.5 vs 64.9; 31.2 vs 63.3 | Helped |
| Use Tucker2-ALS-Sigma instead of FW-LRA | ResNet18 · ResNet18 compression rates 1.4, 1.5, 1.63, and 1.77 | accuracy: 36.7 vs 66.8; 18.9 vs 66.1; 12.2 vs 64.9; 4.2 vs 63.3 | Helped |
| Use CP-ALS-Sigma instead of Greedy Tensor Deflation | ResNet18 · ImageNet Top-1 accuracy at VBMF ratios 0.8, 0.85, and 0.9 | Top-1 accuracy: 35.2 vs 67.9; 20.9 vs 66.5; 5.1 vs 63.1 | Helped |
| Use CP-ALS-Sigma instead of Greedy Tensor Deflation | ResNet18 · ResNet18 convolution layers | relative reconstruction error: 0.195 vs 0.053; 0.277 vs 0.118; 0.217 vs 0.094; 0.207 vs 0.070; 0.159 vs 0.063; 0.292 vs 0.051 | Helped |
| Use Tucker2-ALS-Sigma + SVD-Sigma instead of Tucker2-ALS + SVD | AlexNet · AlexNet, VBMF ratio 0.6 and compression rate 1.46 | Top-1 accuracy: 42.4 vs 49.8 | Helped |
| Compute Sigma from CIFAR-100 rather than CIFAR-10 | Tucker2-ALS-Sigma · ResNet50 compression on CIFAR-10-trained model | accuracy | No clear effect |
| Compute Sigma from an ImageNet training subset rather than CIFAR-10 or CIFAR-100 | Tucker2-ALS-Sigma · CIFAR-10-trained models | accuracy | Hurt |
| Compute Sigma from FGVC-Aircraft rather than the original training data | Tucker2-ALS-Sigma · ImageNet models | accuracy | Helped |
Surprising findings
- Sigma matrix computed from a different dataset (CIFAR-100 or FGVC-Aircraft) gives performance close to using the training dataset, and for ResNet50 CIFAR-100 Sigma even outperformed CIFAR-10 Sigma.
- Using an ImageNet subset for Sigma gave less effective compression on CIFAR-trained models, attributed to image resolution differences.
Limitations
- Only the layer-local functional norm is used; the downstream network q is ignored in the objective.
- Rank selection still relies on isotropic VBMF heuristic rather than the functional norm.
- No performance guarantee without evaluation on data; rank chosen heuristically.
Experimental setup
Datasets: ImageNet CIFAR-10 CIFAR-100 FGVC-Aircraft
Benchmarks: ImageNet CIFAR-10 CIFAR-100 FGVC-Aircraft
Metrics: top-1 accuracy relative reconstruction error (Sigma norm) compression rate
Baselines: Tucker2-ALS CP-ALS Tucker2-ALS with fine-tuning CP-ALS with fine-tuning Greedy tensor deflation FW-LRA AG-LRA
Builds on: Tensorly CP/Tucker ALS VBMF rank selection MUSCO Tucker-2 convolution decomposition data norm of Eckart-Young-type compression (Hsu et al.)
Closest papers by meaning
- Exploring Unexplored Tensor Network Decompositions for Convolutional Neural Networks (NeurIPS 2019 · similarity 0.91)
- Convolutions and More as Einsum: A Tensor Network Perspective with Advances for Second-Order Methods (NeurIPS 2024 · similarity 0.88)
- Effective Interplay between Sparsity and Quantization: From Theory to Practice (ICLR 2025 · similarity 0.88)
- Geometry-aware training of factorized layers in tensor Tucker format (NeurIPS 2024 · similarity 0.87)
- Coarse-To-Fine Tensor Trains for Compact Visual Representations (ICML 2024 · similarity 0.87)
- Robustly overfitting latents for flexible neural image compression (NeurIPS 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.