Research map / Large language models / Language, safety and interpretability
Compressed Sensing for Capability Localization in Large Language Models
Anna Bair, Yixuan Even Xu, Mingjie Sun, J Zico Kolter
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| ablate top-5 compressed-sensing math heads | Llama-3.1-8B · GSM8K | accuracy change: −40.6 ± 6.8 on GSM8K vs −1.0 ± 0.8 on general benchmarks | Hurt |
| ablate top-5 heads on larger vs smaller models | Llama and Qwen models · task-specific benchmarks | accuracy change | Mixed |
| use stratified instead of Bernoulli measurement matrix (ablation) | compressed sensing head identification · GSM8K (Llama-3.1-8B, 100 vs 200 evals) | accuracy drop after ablation: −39.5 (CSB, 200 evals) -> −48.4 (CSS, 100 evals) | Improved |
| replace greedy search with compressed sensing | head identification · GSM8K (Llama-3.1-8B) | accuracy drop and number of evaluations: −55.4 with 5120 evals -> −48.4 with 100 evals | Mixed |
| replace one-shot greedy with stratified compressed sensing | head identification · Swear (Llama-3.1-8B) | accuracy drop: −59.9 -> −85.4 | Improved |
| increase number of evaluations from 100 to 300 (ablation) | compressed sensing · GSM8K (Llama-3.1-8B) | accuracy drop: −40.6 -> −52.4 | Improved |
| ablate more than five heads (ablation) | math heads · GSM8K and Arithmetic (Llama-3.1-8B) | accuracy | No clear effect |
| transfer identified heads across datasets of same capability | math heads · Arithmetic (Llama-3.1-8B) | accuracy change: −60.2 ± 4.3 on Arithmetic from GSM8K-identified heads | Improved |
| ablate universal heads (e.g. L1H29) | Llama-3.1-8B · Math, code and general benchmarks | accuracy change: L1H29: −81.3 math, −63.4 code, −37.9 general | Hurt |
| apply method to WMDP subtasks | Llama-3.1-8B · WMDP | accuracy | No clear effect |
| compare compressed sensing 300 evals to ProxySPEX 500 evals | head identification · GSM8K (Llama-3.1-8B) | accuracy drop: −41.9 vs −41.1 | No clear effect |
Closest papers by meaning
- Can Compressed LLMs Truly Act? An Empirical Evaluation of Agentic Capabilities in LLM Compression (ICML 2025 · similarity 0.88)
- Remarkable Robustness of LLMs: Stages of Inference? (NeurIPS 2025 · similarity 0.88)
- SoLA: Leveraging Soft Activation Sparsity and Low-Rank Decomposition for Large Language Model Compression (AAAI 2025 · similarity 0.87)
- Beyond Linear Approximations: A Novel Pruning Approach for Attention Matrix (ICLR 2025 · similarity 0.86)
- Systematic Outliers in Large Language Models (ICLR 2025 · similarity 0.86)
- Talking Heads: Understanding Inter-Layer Communication in Transformer Language Models (NeurIPS 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
