Research map / Architectures and efficiency / Pruning and efficient tuning
Singular Value Scaling: Efficient Generative Model Compression via Pruned Weights Refinement
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Singular Value Scaling (SVS), a one-time square-root rescaling of pruned weights' singular values that narrows the singular value gap before fine-tuning, improving compressed StyleGAN2, StyleGAN3, and DDPM models.
Problem
Pruned generative models (GANs and diffusion models) often fine-tune slowly and can end up worse than random initialization, because dominant singular vectors in pruned weights dominate training and limit exploration.
Approach
Refine pruned weights at initialization by applying a square-root function to their singular values, shrinking the gap between largest and smallest singular values while keeping singular vectors and their order; bias is rescaled accordingly. The method is architecture-agnostic and applies to both GANs and diffusion models.
Key results
- On StyleGAN2 (small) FFHQ at 70% channel sparsity, DCP-GAN (scaled) reaches FID 5.68/6.60 versus 6.51 (DCP-GAN) and 6.70 (StyleKD) in the base/small settings.
- On DDPM CIFAR-10 with 30% channel sparsity, Diff-Prune (scaled) achieves FID 5.14 vs 5.49 for Diff-Prune; at 70% sparsity 9.38 vs 9.87.
- On StyleGAN3 FFHQ, DCP-GAN (scaled) improves FID from 9.88 to 8.38.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Apply SVS to DCP-GAN-pruned weights | StyleGAN2 · FFHQ and Church, base and small architectures | FID: 6.51 -> 5.68; 8.17 -> 6.60; 4.80 -> 4.62; 6.30 -> 5.18 | Helped |
| Apply SVS to Diff-Prune-pruned weights | DDPM · CIFAR-10 and CelebA-HQ at 30%, 50%, and 70% channel sparsity | FID: 5.49 -> 5.14; 7.77 -> 7.41; 9.87 -> 9.38; 6.17 -> 5.57; 5.32 -> 4.66; 6.30 -> 5.04 | Helped |
| Apply SVS to CAGAN-pruned weights (ablation) | StyleGAN2 small · FFHQ | FID: 9.23 -> 6.83 | Helped |
| Replace square-root scaling with log(x+1) or |log(x)| scaling (ablation) | DCP-GAN-pruned StyleGAN2 small · FFHQ and Church | FID: FFHQ: 6.60 -> 6.18; 6.60 -> 6.32; Church: 5.18 -> 5.30; 5.18 -> 5.43 | Mixed |
| Normalize singular values to 1 (ablation) | DCP-GAN-pruned StyleGAN2 small · FFHQ | FID | Hurt |
| Spectrally normalize singular values (ablation) | DCP-GAN-pruned StyleGAN2 small · FFHQ | FID | Hurt |
| Square singular values instead of scaling them with SVS (ablation) | DCP-GAN-pruned StyleGAN2 small · FFHQ | FID | Hurt |
| Apply spectral normalization to the generator (ablation) | DCP-GAN-pruned StyleGAN2 small · FFHQ | FID: 6.60 -> 8.00 | Hurt |
| Repeat weight refinement during training rather than applying it once at initialization (ablation) | DCP-GAN-pruned StyleGAN2 small · FFHQ | FID: 6.60 -> 8.40 | Hurt |
| Use He initialization for randomly initialized weights (ablation) | StyleGAN2 small · FFHQ and LSUN Church | FID: FFHQ: 6.60 -> 5.84; LSUN Church: 5.18 -> 5.45 | Mixed |
| Increase channel sparsity from 70% to 90% (ablation) | StyleGAN2 small with DCP-GAN and SVS · FFHQ-256 | FID: DCP-GAN: 21.14; DCP-GAN (scaled): 17.60 | Mixed |
| Increase DDPM channel sparsity to 90% (ablation) | DDPM · DDPM compression | FID | Hurt |
Surprising findings
- Pruned-weight initialization can underperform random initialization during fine-tuning, with the gap worsening as model capacity decreases.
- Setting singular values to 1 (orthogonal init) or widening singular value gaps causes training divergence, while the square-root scaling works.
Limitations
- SVS offers mixed results on DDPM SSIM, where it is slightly lower than Diff-Prune.
- Random initialization with He init outperforms pruned weights on StyleGAN2 FFHQ, indicating the benefit is context-dependent.
- Performance degrades at 90% channel sparsity for DDPM, where FID fails to converge.
Experimental setup
Datasets: FFHQ LSUN Church CIFAR-10 CelebA-HQ
Benchmarks: FFHQ-256 LSUN Church-256 CIFAR-10 32x32 CelebA-HQ 64x64
Metrics: FID precision recall density coverage SSIM MACs params
Baselines: StyleKD DCP-GAN CAGAN GS Diff-Prune
Builds on: DCP-GAN Diff-Prune StyleKD CAGAN
Closest papers by meaning
- Content-Aware GAN Compression (CVPR 2021 · similarity 0.90)
- RG-GAN: Dynamic Regenerative Pruning for Data-Efficient Generative Adversarial Networks (AAAI 2024 · similarity 0.89)
- Compressing Image-to-Image Translation GANs Using Local Density Structures on Their Learned Manifold (AAAI 2024 · similarity 0.88)
- GAN Compression: Efficient Architectures for Interactive Conditional GANs (CVPR 2020 · similarity 0.88)
- Diversity-aware Channel Pruning for StyleGAN Compression (CVPR 2024 · similarity 0.87)
- Direct Discriminative Optimization: Your Likelihood-Based Visual Generative Model is Secretly a GAN Discriminator (ICML 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.