Research map / Architectures and efficiency / Pruning and efficient tuning
Pivoting Factorization: A Compact Meta Low-Rank Representation of Sparsity for Efficient Inference in Large Language Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes PIFA, a lossless pivot-row-based meta representation that compresses any low-rank factorization, plus an online error-accumulation-minimizing reconstruction (M), combined as MPIFA for retraining-free LLM low-rank pruning.
Problem
Low-rank pruning of LLMs lags semi-structured pruning in perplexity at similar densities (often ~2x PPL), and low-rank factorizations carry redundancy and reconstruction methods accumulate errors across layers.
Approach
Identify pivot (linearly independent) rows of the low-rank weight product via QR/LU with pivoting and express non-pivot rows as linear combinations, yielding a lossless compact representation (PIFA); pair this with an online least-squares reconstruction whose target mixes dense and low-rank data flows to curb error accumulation.
Key results
- PIFA layer at rank 50% of dimension gives 24.2% additional memory savings and 24.6% faster inference over SVD low-rank layers, losslessly.
- MPIFA on LLaMA2-7B at 50% density reaches WikiText2 PPL 12.77 vs SVD-LLM 27.19 (dense 5.47).
- At 55% density, MPIFA-NS achieves PPL 9.68 on LLaMA2-7B vs 11.40 for Wanda 2:4 and 10.85 for RIA 2:4 under equal memory reduction.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add online error-accumulation-minimization reconstruction to SVD-LLM pruning (ablation) | Low-rank pruning · WikiText2, 50% density, LLaMA2-7B | perplexity: 33.27 -> 16.55 | Helped |
| Add PIFA after online reconstruction (ablation) | W + M · WikiText2, 50% density, LLaMA2-7B | perplexity: 16.55 -> 12.77 | Helped |
| Use full-batch reconstruction after SVD-LLM pruning (ablation) | LLaMA2-7B · WikiText2, 90% density | perplexity: 7.27 -> 7.60 | Hurt |
| Use mix ratio λ = 0.25 instead of λ = 0 (ablation) | Online reconstruction in MPIFA · WikiText2, 0.5 density | perplexity | Helped |
| Increase the mix ratio excessively (ablation) | Online reconstruction in MPIFA · Calibration-data evaluation | perplexity | Hurt |
| Increase the number of calibration samples (ablation) | MPIFA · WikiText2 | perplexity | Helped |
| Reconstruct both U and Vᵀ rather than only one factor (ablation) | MPIFA · WikiText2, sufficient calibration samples | perplexity | Helped |
| Apply MPIFA_NS 55% instead of RIA 2:4 | LLaMA2-7B · WikiText2, same memory reduction | perplexity: 10.85 -> 9.68 | Helped |
| Apply MPIFA_NS 55% instead of RIA 2:4 | LLaMA2-13B · WikiText2, same memory reduction | perplexity: 8.03 -> 7.93 | Helped |
| Use PIFA 55% instead of 2:4 semi-sparse kernels | Linear layer inference · A6000, dimension 32768 | speedup: 2.10× vs 0.79× | Helped |
| Use MPIFA_NS 55% instead of dense inference | LLaMA2-7B · End-to-end, A6000, KV cache enabled | throughput (token/s): 3409 -> 4840 | Helped |
Surprising findings
- Full-batch reconstruction (W+U) occasionally worsens perplexity relative to pruning alone, attributed to overfitting to calibration data.
- Semi-structured 2:4 implementations in torch are slower than dense layers at some dimensions and fail with KV cache enabled.
Limitations
- PIFA does not reduce matrix rank by itself and must be combined with a low-rank compression method.
- Reconstruction of V^T can be ill-conditioned with few calibration samples, requiring regularization.
- Fine-tuning and some baselines (ASVD on LLaMA2-70B) run out of memory or require substantial compute.
Experimental setup
Datasets: WikiText2 C4
Benchmarks: WikiText2 C4 SuperGLUE LLM inference throughput
Metrics: perplexity zero-shot accuracy layerwise speedup memory usage throughput
Baselines: SVD ASVD SVD-LLM Magnitude 2:4 Wanda 2:4 RIA 2:4 LLM-Pruner
Builds on: SVD-LLM ASVD OWL
Closest papers by meaning
- Targeted Low-rank Refinement: Enhancing Sparse Language Models with Precision (ICML 2025 · similarity 0.89)
- Compressing Large Language Models using Low Rank and Low Precision Decomposition (NeurIPS 2024 · similarity 0.89)
- SliceGPT: Compress Large Language Models by Deleting Rows and Columns (ICLR 2024 · similarity 0.88)
- SoLA: Leveraging Soft Activation Sparsity and Low-Rank Decomposition for Large Language Model Compression (AAAI 2025 · similarity 0.88)
- Discovering Sparsity Allocation for Layer-wise Pruning of Large Language Models (NeurIPS 2024 · similarity 0.87)
- Týr-the-Pruner: Structural Pruning LLMs via Global Sparsity Distribution Optimization (NeurIPS 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.