Research map / Architectures and efficiency / Pruning and efficient tuning
Compress to Impress: Efficient LLM Adaptation Using a Single Gradient Step on 100 Samples
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Speeds up LASER-style training-free rank reduction by ranking matrices with a single gradient step on singular values using 100 samples, and adds multi-subspace (block-split) factorization, yielding up to 52x speedup and up to 24.6-point accuracy gains on GPT-J and RoBERTa.
Problem
LASER's exhaustive per-matrix, per-rate layer search requires full-dataset forward passes, making post-training rank-reduction adaptation of LLMs impractical for rapid deployment.
Approach
Score weight matrices by the gradient of their singular values from one backward pass on ~100 examples, and factor row-clusters (block-split into K consecutive row blocks) with separate SVDs, enlarging the factorization search space to remove overfitting noise, while using 100 examples for both scoring and evaluation.
Key results
- Clustering LASER with 100 gradients and 100-point evaluation (CL-100G-100E) on GPT-J averages 52.0x speedup over LASER with +0.95 points average change from LASER.
- On GPT-J, Clustering LASER with full search improves over LASER on all eight datasets (e.g., BigBench-Epistemic Reasoning 38.3 to 62.9).
- On Roberta, CL-100G-100E gives about 22.2x average speedup while staying near LASER accuracy (-0.40 points average change).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add multi-subspace clustering to LASER (ablation) | GPT-J · CounterFact | accuracy: 24.0 -> 24.5 | Helped |
| Add multi-subspace clustering to LASER (ablation) | GPT-J · HotPotQA | accuracy: 19.5 -> 20.3 | Helped |
| Add multi-subspace clustering to LASER (ablation) | GPT-J · FEVER | accuracy: 56.2 -> 57.8 | Helped |
| Add multi-subspace clustering to LASER (ablation) | GPT-J · Bios Gender | accuracy: 97.5 -> 97.7 | Helped |
| Add multi-subspace clustering to LASER (ablation) | GPT-J · Bios Profession | accuracy: 82.1 -> 82.3 | Helped |
| Add multi-subspace clustering to LASER (ablation) | GPT-J · TruthfulQA | accuracy: 55.6 -> 56.1 | Helped |
| Add multi-subspace clustering to LASER (ablation) | GPT-J · BigBench–Epistemic Reasoning | accuracy: 38.3 -> 62.9 | Helped |
| Add multi-subspace clustering to LASER (ablation) | GPT-J · BigBench–WikidataQA | accuracy: 65.9 -> 66.5 | Helped |
| Use 100-example evaluation instead of standard evaluation (ablation) | Clustering LASER with 100-example gradient scoring · GPT-J benchmarks | accuracy | Mixed |
| Use gradient-guided matrix selection and 100-example gradient scoring (ablation) | LASER · GPT-J benchmarks | accuracy | Mixed |
| Use multi-subspace clustering with 100-example gradients and evaluation | GPT-J adaptation · GPT-J benchmarks | speedup: 52.0x | Helped |
| Replace consecutive row-block splitting with optimized projective clustering (ablation) | Clustering LASER on RoBERTa · FEVER | accuracy: 52.7% -> 53.5% | Helped |
Surprising findings
- Adding attention matrices (k_proj, q_proj) to the search typically reverted GPT-J to baseline accuracy, so only MLP matrices were kept.
- Singular-value gradient diagonal entries showed essentially no correlation with singular value magnitudes (Spearman rho around 0.01-0.035).
Limitations
- Experiments limited to small, English-only models (GPT-J, RoBERTa).
- Inherits LASER's finite candidate search without gradient-based weight updates.
- Block-splitting heuristic replaces optimal projective clustering; optimal clustering gave little gain (only FEVER changed).
Experimental setup
Datasets: CounterFact HotPotQA FEVER Bias in Bios TruthfulQA BigBench-Epistemic Reasoning BigBench-WikidataQA
Benchmarks: CounterFact HotPotQA FEVER Bios Gender Bios Profession TruthfulQA BigBench-Epistemic Reasoning BigBench-WikidataQA
Metrics: accuracy speedup
Baselines: LASER Baseline (unmodified model)
Builds on: LASER LoRA
Closest papers by meaning
- Memory-Efficient LLM Training with Online Subspace Descent (NeurIPS 2024 · similarity 0.89)
- SLoPe: Double-Pruned Sparse Plus Lazy Low-Rank Adapter Pretraining of LLMs (ICLR 2025 · similarity 0.89)
- From Low Rank Gradient Subspace Stabilization to Low-Rank Weights: Observations, Theories, and Applications (ICML 2025 · similarity 0.89)
- ESPACE: Dimensionality Reduction of Activations for Model Compression (NeurIPS 2024 · similarity 0.88)
- S$^{2}$FT: Efficient, Scalable and Generalizable LLM Fine-tuning by Structured Sparsity (NeurIPS 2024 · similarity 0.87)
- Rethinking LLM Evaluation: Can We Evaluate LLMs with 200× Less Data? (ICLR 2026 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.