Research map / Architectures and efficiency / RNNs and normalization
Swift-SVD: Theoretical Optimality Meets Practical Efficiency in Low-Rank LLM Compression
Ruoling Qi, Yirui Liu, Xuaner Wu, Xiangyu Wang, Ming Li, Chen Chen, Jian Chen, Yin Chen, Qizhen Weng
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add dynamic rank allocation (loss plus layer importance with preserved ratio) (ablation) | Swift-SVD · C4 PPL, LLaMA-7B ratio 0.6 | perplexity: 23.32 -> 21.92 | Improved |
| add dynamic rank allocation (ablation) | Swift-SVD · LLaMA-7B zero-shot average accuracy ratio 0.8 | accuracy: 0.50 -> 0.51 | Improved |
| allocate ranks by Frobenius loss only, no preserved ratio (ablation) | Swift-SVD · C4 PPL, LLaMA-7B ratio 0.8 | perplexity: 11.42 -> 16.04 | Hurt |
| allocate ranks by layer importance only, no preserved ratio (ablation) | Swift-SVD · C4 PPL, LLaMA-7B ratio 0.8 | perplexity: 11.42 -> 14.88 | Hurt |
| add fixed preserved ratio 0.5 to loss-only allocation (ablation) | Swift-SVD(C) · C4 PPL, LLaMA-7B ratio 0.8 | perplexity: 16.04 -> 11.78 | Improved |
| add fixed preserved ratio 0.5 to importance-only allocation (ablation) | Swift-SVD(I) · C4 PPL, LLaMA-7B ratio 0.8 | perplexity: 14.88 -> 11.73 | Improved |
| combine loss and importance in allocation instead of importance alone with preserved ratio (ablation) | Swift-SVD-dagger(I) · C4 PPL, OPT-6.7B ratio 0.8 | perplexity: 14.74 -> 13.68 | Improved |
| use Swift-SVD* dynamic allocation instead of Dobi-SVD training-based dynamic allocation | Dobi-SVD · C4 PPL, LLaMA-7B ratio 0.4 | perplexity: 190.62 -> 137.77 | Improved |
| calibrate on C4 only and evaluate on WikiText-2 | Swift-SVD · LLaMA-7B ratio 0.6 nsamples 256 | perplexity: 13.42 -> 37.00 | Hurt |
| increase calibration samples from 256 to 2048 (ablation) | Swift-SVD · C4 PPL, LLaMA-7B ratio 0.6 | perplexity: 23.17 -> 22.37 | Improved |
| use Swift-SVD closed-form eigendecomposition instead of SVD-LLM | SVD-LLM · reconstruction loss [4096x4096]^2 | loss: 3365.6499 -> 3308.6428 | Improved |
| use Swift-SVD instead of Dobi-SVD(w/o) for compression, 512 samples | Dobi-SVD(w/o) · compression time | speedup: 1.0x -> 76.9x | Improved |
Closest papers by meaning
- LowRankArena: A Standardized Evaluation Platform for SVD-Based LLM Compression (NeurIPS 2026 · similarity 0.94)
- Dobi-SVD: Differentiable SVD for LLM Compression and Some New Perspectives (ICLR 2025 · similarity 0.92)
- Basis Sharing: Cross-Layer Parameter Sharing for Large Language Model Compression (ICLR 2025 · similarity 0.91)
- SVD-LLM: Truncation-aware Singular Value Decomposition for Large Language Model Compression (ICLR 2025 · similarity 0.91)
- Compressing Large Language Models using Low Rank and Low Precision Decomposition (NeurIPS 2024 · similarity 0.89)
- 3BASiL: An Algorithmic Framework for Sparse plus Low-Rank Compression of LLMs (NeurIPS 2025 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
