Research map / Architectures and efficiency / Pruning and efficient tuning
PAT: Pruning-Aware Tuning for Large Language Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Pruning-Aware Tuning (PAT), which structurally prunes LLM hidden dimensions during LoRA-style fine-tuning using Hybrid Sparsification Modules, a unified trainable mask, and an Identity Loss.
Problem
Structural pruning of LLMs usually causes large accuracy loss, and post-hoc pruning before or after fine-tuning recovers poorly; the paper asks how to prune while fine-tuning to preserve performance.
Approach
Insert Hybrid Sparsification Modules (identity plus low-rank Hybrid-Identity-Operator) between Attention and FFN blocks, govern all of them by one globally shared trainable mask, exploit RMSNorm's zero-preservation to prune channels, and add an Identity Loss that decouples rotation and scaling of the HSMs, so pruning and fine-tuning happen jointly.
Key results
- Llama2-7B with 25% pruning under PAT reaches 60.02 average accuracy versus 58.76 for unpruned LoRA-64 (+1.26%).
- At 30% pruning, PAT yields about 1.33x speedup on average for Llama2 models, and Llama2-13B pruned models run where the base model OOMs above batch size 288 on A100-80GB.
- At 30% pruning, PAT on Gemma 7B scores 64.58 versus 71.59 for LoRA-64 and outperforms LLM-Pruner and SliceGPT pruned baselines at every ratio.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use PAT rather than post-pruning fine-tuning or fine-tune-then-prune | Llama2-7B at 25% pruning · 14-task zero-shot average | accuracy: 52.50 -> 60.02 | Helped |
| Use PAT rather than post-pruning fine-tuning or fine-tune-then-prune | Llama2-13B at 20% pruning · 14-task zero-shot average | accuracy: 65.28 -> 69.37 | Helped |
| Use PAT rather than post-pruning fine-tuning or fine-tune-then-prune | Yi-1.5-34B at 30% pruning · 14-task zero-shot average | accuracy: 71.81 -> 77.89 | Helped |
| Add a trainable mask instead of using a frozen mask (ablation) | PAT, Llama2-7B · 14-task zero-shot average, 25% pruning | accuracy: 54.97 -> 60.02 | Helped |
| Add Identity Loss (ablation) | PAT, Llama2-7B · 14-task zero-shot average, 25% pruning | accuracy: 58.62 -> 60.02 | Helped |
| Add a trainable mask instead of using a frozen mask (ablation) | PAT, Llama2-7B · 14-task zero-shot average, 30% pruning | accuracy: 52.72 -> 57.81 | Helped |
| Add Identity Loss (ablation) | PAT, Llama2-7B · 14-task zero-shot average, 30% pruning | accuracy: 56.59 -> 57.81 | Helped |
| Use 30% pruning rather than no pruning with LoRA-64 | PAT, Llama2-7B · 14-task zero-shot average | accuracy: 58.76 -> 57.81 | Hurt |
| Use 25% pruning rather than no pruning with LoRA-64 | PAT, Llama2-13B · 14-task zero-shot average | accuracy: 66.74 -> 66.58 | Mixed |
Surprising findings
- Llama2 7B and 13B under PAT at moderate pruning ratios (below 30% and 20%, respectively) exceed the unpruned LoRA baseline.
- Removing the trainable mask (using a frozen mask) costs up to 4.06% accuracy on Llama2-13B at 30% pruning.
Limitations
- Accuracy drops noticeably for some models (e.g., Gemma 7B at 20% pruning and Gemma 2B), and the pruning feasibility varies by model.
- Evaluation is limited to zero-shot accuracy with a first-capital-word metric and a single fine-tuning dataset subset.
- Hyperparameters such as HIO rank were tuned per model to match LoRA-64 parameter count.
Experimental setup
Datasets: LaMini-instruction
Benchmarks: ARC-Challenge ARC-Easy BoolQ COPA HellaSwag MMLU MultiRC OpenBookQA PIQA RTE SIQA WIC
Metrics: zero-shot accuracy (first-capital-word) average accuracy VRAM usage latency speedup
Baselines: LoRA-64 LLM-Pruner SliceGPT DoRA-64
Builds on: LoRA DoRA SliceGPT LLM-Pruner RMSNorm
Closest papers by meaning
- APT: Adaptive Pruning and Tuning Pretrained Language Models for Efficient Training and Inference (ICML 2024 · similarity 0.93)
- SPP: Sparsity-Preserved Parameter-Efficient Fine-Tuning for Large Language Models (ICML 2024 · similarity 0.92)
- Layer as Puzzle Pieces: Compressing Large Language Models through Layer Concatenation (NeurIPS 2025 · similarity 0.91)
- An Efficient Pruner for Large Language Model with Theoretical Guarantee (ICML 2025 · similarity 0.91)
- BaWA: Automatic Optimizing Pruning Metric for Large Language Models with Balanced Weight and Activation (ICML 2025 · similarity 0.91)
- A Simple and Effective Pruning Approach for Large Language Models (ICLR 2024 · similarity 0.91)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.