Research map / Architectures and efficiency / Pruning and efficient tuning
MoDeGPT: Modular Decomposition for Large Language Model Compression
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes MoDeGPT, a training-free structured LLM compression method that jointly decomposes module-level matrix pairs (MLP, key-query, value-output) using Nyström, CR, and SVD approximations, without backprop or extra adapters.
Problem
LLMs are too computationally and memory-intensive for resource-limited devices; existing low-rank compression either discards many ranks or adds substantial parameter overhead (e.g., adapters in SliceGPT), and gradient-based structured pruning is costly.
Approach
Group consecutive weight matrices into Transformer modules (MLP up/down, query-key, value-output) and compress each module jointly via closed-form matrix decompositions (Nyström for Type-I, CR for Type-II, SVD for Type-III) with reconstruction error bounds, plus a one-pass global sparsity allocation based on Block Influence scores.
Key results
- On LLaMA-2 7B at 20% compression, MoDeGPT perplexity rises about 20% (vs ~50% for next best), per the paper's comparison.
- On LLaMA-2 70B at 30% compression, MoDeGPT + our sparsity + Alpaca reaches 66.79 average zero-shot accuracy vs 70.02 dense (about 3.2% drop).
- MoDeGPT increases inference throughput by up to 46% (58% at 50% compression on LLaMA-2 7B, A100, batch 256) and cuts compression cost by 97% for 7B and 98.5% for 13B versus LLM Surgeon.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Calibrating MoDeGPT with Alpaca instead of WikiText-2 (ablation) | MoDeGPT on LLaMA-2 7B · Zero-shot 30% and 40% compression | average zero-shot accuracy: 60.78 to 62.02 at 30%; 51.88 to 57.58 at 40% | Helped |
| Global sparsity allocation (entropic, BI-score based) instead of OWL sparsity (ablation) | MoDeGPT on LLaMA-2 70B · 30% compression zero-shot | average accuracy: 65.51 (ours) vs 63.08 (OWL); OWL has lower perplexity-free ranking, ours has higher perplexity 4.89 vs 4.67 | Mixed |
| Replacing SVD-based compression with MoDeGPT modular decomposition | LLaMA-2 7B and 13B at all tested sparsities · WikiText-2 perplexity | perplexity: SVD perplexity overflows or reaches thousands vs MoDeGPT 5.48-11.88 at 10-50% | Helped |
| Gradient-free MoDeGPT vs gradient-based LLM Surgeon | LLaMA-2 7B at 30% compression · zero-shot average | average accuracy: 60.78 vs 59.03 (WikiText-2 calibration); 62.02 with Alpaca | Helped |
| Gradient-free MoDeGPT vs gradient-based LLM Surgeon on perplexity | LLaMA-2 7B at 20-50% compression · WikiText-2 perplexity | perplexity: MoDeGPT 6.16 vs LLM Surgeon 6.18 at 20% (near tie); LLM Surgeon 7.83 vs 7.51 at 30%, LLM Surgeon better at 10% (5.25 vs | Hurt |
| Alpaca calibration with MoDeGPT on LLaMA-3 8B | LLaMA-3 8B at 25% compression · zero-shot average | average accuracy: 63.96 vs 46.17 for SliceGPT-Alpaca and 43.24 for ShortGPT-Alpaca | Helped |
Surprising findings
- Type-II (query-key) modules are the most sensitive to compression once normalized by parameter count, despite the MLP causing most perplexity degradation in absolute terms.
- Calibrating on Alpaca instead of WikiText-2 substantially improved zero-shot performance for compressed LLaMA-2 models.
Limitations
- Key-query module compression can slow generation at compression rates below 10% due to head-dependent index selection overhead.
- Compression of LLaMA-3 8B with GQA requires modified algorithms, and the performance gap to baselines is reported in limited settings.
- Perplexity at 70B with our sparsity is slightly higher than OWL-based sparsity allocation.
Experimental setup
Datasets: WikiText-2 Alpaca
Benchmarks: WikiText-2 ARC-e ARC-c PIQA WinoGrande HellaSwag OpenBookQA MathQA BoolQ COPA MMLU LAMBADA
Metrics: perplexity zero-shot accuracy average accuracy throughput (tokens/sec) compression cost (GPU hours)
Baselines: SliceGPT ShortGPT SLEB LLM-Pruner LLM Surgeon OWL SVD Magnitude Pruning Uniform Pruning K-OBD
Builds on: SliceGPT LLM Surgeon Nyström approximation CR decomposition Block Influence score (ShortGPT)
Closest papers by meaning
- SliceGPT: Compress Large Language Models by Deleting Rows and Columns (ICLR 2024 · similarity 0.91)
- Compressing Large Language Models using Low Rank and Low Precision Decomposition (NeurIPS 2024 · similarity 0.91)
- 3BASiL: An Algorithmic Framework for Sparse plus Low-Rank Compression of LLMs (NeurIPS 2025 · similarity 0.91)
- SoLA: Leveraging Soft Activation Sparsity and Low-Rank Decomposition for Large Language Model Compression (AAAI 2025 · similarity 0.91)
- Search for Efficient Large Language Models (NeurIPS 2024 · similarity 0.89)
- EvoPress: Accurate Dynamic Model Compression via Evolutionary Search (ICML 2025 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.