Research map / Architectures and efficiency / Pruning and efficient tuning
SliceGPT: Compress Large Language Models by Deleting Rows and Columns
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes SliceGPT, a post-training structured pruning method that uses computational invariance under orthogonal transformations and PCA to delete rows and columns of weight matrices, reducing the embedding dimension of LLMs.
Problem
Large language models are costly to run in compute and memory; existing sparsification methods need extra data structures or give limited real speedup on current hardware, and many structured pruning methods require expensive recovery fine-tuning.
Approach
Exploit computational invariance in RMSNorm-connected transformers: insert orthogonal transforms per block (computed by PCA on calibration activations) that leave the network unchanged, then slice off the minor principal components so weight matrices and the residual stream become smaller dense matrices.
Key results
- On LLaMA-2 70B with 25% slicing, zero-shot average retains about 96% of dense performance (69.75 vs 76.57 without RFT); with RFT on Alpaca it reaches 74.3 at 30% slicing vs 76.6 dense.
- On WikiText-2, SliceGPT at 25% gives 4.60 PPL on LLaMA-2 70B versus 4.98 for SparseGPT 2:4 and 3.32 dense.
- On RTX6000 24GB GPUs, 25%-sliced LLaMA-2 70B reduces per-token inference compute to 64% of dense (1075 vs 1764 GPUms).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| slice 25% of embedding dimensions | Llama 70B · WikiText-2 | perplexity: 4.98 -> 4.60 | Helped |
| slice 30% of embedding dimensions | OPT 66B · WikiText-2 | perplexity: 10.22 -> 9.85 | Helped |
| slice 25% of embedding dimensions | Llama 70B · Zero-shot average accuracy, Alpaca calibration | accuracy: 76.57 -> 73.59 | Hurt |
| apply recovery fine-tuning on Alpaca after 30% slicing | Llama 70B · Zero-shot average accuracy | accuracy: 66.11 -> 74.3 | Helped |
| apply recovery fine-tuning on Alpaca after 25% slicing | Phi-2 · Zero-shot average accuracy | accuracy: 54.46 -> 65.24 | Helped |
| use FP32 instead of double precision for PCA (ablation) | SliceGPT · WikiText-2 perplexity, Llama 70B at 25% slicing | perplexity: 4.60 -> 7.01 | Hurt |
| reduce calibration set size from 1024 to 128 (ablation) | SliceGPT · WikiText-2 perplexity, Llama 70B at 25% slicing | perplexity: 4.60 -> 4.89 | Hurt |
| vary slicing level by layer instead of using constant 25% slicing (ablation) | OPT 66B · WikiText-2 | perplexity: 9.75 -> 9.63 | Helped |
| vary slicing level by layer instead of using constant 25% slicing (ablation) | Llama 70B · WikiText-2 | perplexity: 4.44 -> 4.63 | Hurt |
| slice 25% of embedding dimensions | Llama 70B · Single-token inference on A100 (40GB) | total compute: 500 GPUms -> 330 GPUms | Helped |
| slice 50% of embedding dimensions | OPT 66B · Tokens/s on H100 80GB | throughput: 141 -> 441 (6.26×) | Helped |
Surprising findings
- Varying the slicing level by layer improves OPT perplexity but worsens LLaMA perplexity.
- A simple norm-based column/row pruning baseline gave WikiText-2 perplexity in the thousands, so the PCA-based transformation is essential.
Limitations
- Perplexity is worse than SparseGPT 2:4 for smaller models and at 50% slicing.
- Zero-shot accuracy drops notably for LLaMA and Phi-2 without recovery fine-tuning, and OPT did not benefit from RFT.
- End-to-end speedups are measured with Hugging Face inference without continuous batching or sharding; sparse baselines lacked end-to-end speedup.
Experimental setup
Datasets: WikiText-2 Alpaca
Benchmarks: WikiText-2 PIQA WinoGrande HellaSwag ARC-e ARC-c
Metrics: perplexity zero-shot accuracy throughput (tokens/s) per-token inference time (GPUms)
Baselines: SparseGPT 2:4 LLM-Pruner magnitude pruning
Builds on: SparseGPT LoRA Optimal Brain Surgeon GPTQ
Closest papers by meaning
- MoDeGPT: Modular Decomposition for Large Language Model Compression (ICLR 2025 · similarity 0.91)
- SoLA: Leveraging Soft Activation Sparsity and Low-Rank Decomposition for Large Language Model Compression (AAAI 2025 · similarity 0.89)
- BESA: Pruning Large Language Models with Blockwise Parameter-Efficient Sparsity Allocation (ICLR 2024 · similarity 0.89)
- Determining Layer-wise Sparsity for Large Language Models Through a Theoretical Perspective (ICML 2025 · similarity 0.89)
- SparseLLM: Towards Global Pruning of Pre-trained Language Models (NeurIPS 2024 · similarity 0.89)
- Pruning Large Language Models with Semi-Structural Adaptive Sparse Training (AAAI 2025 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.