Research map / Architectures and efficiency / Pruning and efficient tuning
GEM: A Scale-Aware and Distribution-Sensitive Sparse Fine-Tuning Framework for Effective Downstream Adaptation
Sungmin Kang, Jisoo Kim, Salman Avestimehr, Sunwoo Lee
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add entropy to norm-based layer importance (ablation) | GEM (OPT-1.3B) · MultiRC | validation accuracy: 66.56% -> 70.31% | Improved |
| add entropy to norm-based layer importance (ablation) | GEM (OPT-125M) · SST-2 | accuracy: 87.12 -> 89.53 | Improved |
| use entropy-guided allocation instead of uniform per-layer allocation (ablation) | GEM (Phi-2) · SQuAD | F1: 88.70 -> 90.89 | Improved |
| use norm only (without entropy) layer score instead of uniform (ablation) | GEM (Phi-2) · SST-2 | accuracy: 93.37 -> 92.66 | Hurt |
| use entropy only (no norm) layer score (ablation) | GEM (OPT-1.3B) · SST-2 | accuracy: 91.98 -> 91.92 (vs Uniform) | No clear effect |
| select parameters by gradient-to-weight ratio instead of gradient magnitude (ablation) | GEM (OPT-1.3B, 0.1%) · 7-task average | average score: 71.42 -> 77.47 | Improved |
| select parameters by gradient-to-weight ratio instead of random | GEM (OPT-125M, 0.1%) · 7-task average | average score: 66.29 -> 68.85 | Improved |
| reduce tunable ratio below 0.1% (ablation) | GEM (Phi-2) · 7-task average | average score | Hurt |
| use GEM instead of full fine-tuning | Phi-2, 0.1% params · GSM8k | exact match: 53.41 -> 48.78 | Hurt |
| use GEM instead of full fine-tuning | Phi-2, 0.1% params · MBPP | pass@3: 47.83 -> 43.58 | Hurt |
| use GEM instead of full fine-tuning | OPT-125M/1.3B/Phi-2, 0.1% params · 7-task average | average score: OPT-125M 67.23 -> 68.85; OPT-1.3B 76.96 -> 77.47; Phi-2 85.27 -> 85.52 | Mixed |
| use GEM instead of AdaLoRA | Phi-2 · 7-task average | average score: 84.64 -> 85.52 | Improved |
Closest papers by meaning
- RoSA: Rotational Sparse Adaptation for Memory-Efficient Fine-Tuning (NeurIPS 2026 · similarity 0.89)
- Parameter Efficient Fine-tuning via Explained Variance Adaptation (NeurIPS 2025 · similarity 0.89)
- PACE: Marrying generalization in PArameter-efficient fine-tuning with Consistency rEgularization (NeurIPS 2024 · similarity 0.89)
- Sparse is Enough in Fine-tuning Pre-trained Large Language Models (ICML 2024 · similarity 0.88)
- S$^{2}$FT: Efficient, Scalable and Generalizable LLM Fine-tuning by Structured Sparsity (NeurIPS 2024 · similarity 0.88)
- Sensitivity-Aware Efficient Fine-Tuning via Compact Dynamic-Rank Adaptation (CVPR 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
