Research map / Architectures and efficiency / Pruning and efficient tuning
Enhancing In-Context Learning Performance with just SVD-Based Weight Pruning: A Theoretical Perspective
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Shows SVD-based low-rank weight pruning can improve in-context learning, with deep-layer pruning more stable, explains this via implicit GD trajectories and mutual-information generalization bounds, and proposes a derivative-free pruning search (Algorithm 1).
Problem
Why SVD-based weight pruning enhances in-context learning (ICL) performance and why deep and shallow layers respond differently to pruning, plus how to exploit this for downstream tasks.
Approach
Models ICL in linear attention as implicit gradient descent over layers, derives mutual-information generalization bounds over the full implicit GD trajectory, and uses the resulting view (norm control of implicit gradients, condition numbers) to justify SVD pruning and a greedy layer/clipping-rate search.
Key results
- SVD-based pruning of weights often improves ICL accuracy over the unpruned baseline across layers and modules on SST-2, RTE and COPA with GPT-J-6B and LLaMA2-7B.
- The best performance sometimes occurs at a clipping rate of 0.995, i.e. removing about 99.5% of the matrix rank.
- Algorithm 1 (layer selection by condition number plus greedy clipping-rate search) visibly improves ICL accuracy over plain ICL on the evaluated tasks for both models.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Apply SVD-based weight pruning across layers and module types | GPT-J-6B and Llama2-7B · SST-2, RTE, COPA | ICL performance | Mixed |
| Prune weights in deep layers rather than shallow layers | GPT-J-6B and Llama2-7B · SST-2, RTE, COPA | stability of ICL performance | Helped |
| Sharply reduce weights in shallow layers | GPT-J-6B and Llama2-7B · SST-2, RTE, COPA | ICL performance | Hurt |
| Use Algorithm 1 to select a layer and search for its clipping rate | GPT-J-6B and Llama2-7B · SST-2, AGNEWS, EMOC, MRPC, RTE, CB, COPA | test performance | Helped |
| Reduce the number of ICL shots | SVD-based weight pruning · SST-2 | rate of performance collapse after sharp shallow-layer pruning | Helped |
Surprising findings
- Pruning deep-layer weights can drastically reduce rank without degradation and sometimes improve performance, while shallow-layer pruning causes collapse to chance.
- Removing up to 99.5% of singular values can give the best ICL performance.
Limitations
- Theory is restricted to linear attention without softmax and with simplifying assumptions such as Gaussian gradient noise.
- Cost of searching the optimal clipping rate is not addressed.
- Evaluated only on NLP tasks with two open-source models.
Experimental setup
Datasets: SST-2 RTE COPA AGNEWS EMOC MRPC CB
Benchmarks: SST-2 AGNEWS EMOC MRPC RTE CB COPA
Metrics: accuracy
Baselines: ICL without pruning
Builds on: linear attention as implicit gradient descent information-theoretic generalization bounds Eckart-Young-Mirsky theorem
Closest papers by meaning
- Position: Do pretrained Transformers Learn In-Context by Gradient Descent? (ICML 2024 · similarity 0.90)
- How Do Nonlinear Transformers Learn and Generalize in In-Context Learning? (ICML 2024 · similarity 0.90)
- Vector-ICL: In-context Learning with Continuous Vector Representations (ICLR 2025 · similarity 0.89)
- In-Context Learning State Vector with Inner and Momentum Optimization (NeurIPS 2024 · similarity 0.88)
- Transformers are Minimax Optimal Nonparametric In-Context Learners (NeurIPS 2024 · similarity 0.88)
- Exact Conversion of In-Context Learning to Model Weights in Linearized-Attention Transformers (ICML 2024 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.