Research map / Large language models / Instruction and fine-tuning
Enhancing Large Language Model Performance with Gradient-Based Parameter Selection
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Gradient-Based Mask Tuning (GMT), which masks low-magnitude gradients during backpropagation so only task-relevant parameters are updated, improving SFT and DPO results on code, math, and general-domain benchmarks.
Problem
Full-parameter fine-tuning of LLMs updates many parameters redundantly; existing redundancy-reduction methods either ignore task-specific data (random masking, dropout-style methods) or remove structural components or parameters post hoc, limiting scalability and architecture integrity.
Approach
During training, accumulate gradients over N mini-batches, keep only the top-k percentile of gradient entries by absolute value (element-wise), and update only those parameters, so the mask is task-data-driven and the network architecture is unchanged.
Key results
- On Mistral-7B, the method averages 60.0 on code benchmarks vs 58.8 for SFT and 53.6 vs 51.2 on math; GSM8k improves 3.3 points over SFT.
- On Llama2-13B Tulu V2 SFT, the method averages 50.5 vs 47.2 for SFT and 50.2 for RMT across general benchmarks.
- Training FLOPs are nearly identical to vanilla SFT (about 1.48e18 vs 1.481e18 for Mistral-7B), with training speed about 12% lower than SFT (13.43 vs 15.32 samples/s).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use gradient magnitude to select parameters for updates instead of the random selection in RMT | GMT · code generation, Mistral-7B; average across HumanEval, HumanEval+, MBPP, and MBPP+ | average score: 59.0 -> 60.0 | Helped |
| Use gradient magnitude to select parameters for updates instead of vanilla SFT | GMT · math reasoning, Mistral-7B | GSM8k accuracy: 75.3 -> 78.6 | Helped |
| Use gradient magnitude to select parameters for updates instead of vanilla SFT | GMT · math reasoning, Llama3-8B | GSM8k accuracy: 78.1 -> 82.0 | Helped |
| Use gradient magnitude to select parameters for updates instead of vanilla SFT | GMT · Tulu V2 SFT, Llama2-7B | average score: 41.0 -> 44.0 | Helped |
| Use gradient magnitude to select parameters for updates instead of random parameter selection | GMT · UltraFeedback DPO, Llama2-13B | average score: 51.6 -> 52.3 | Helped |
| Use gradient magnitude to select updates instead of randomly selecting half the parameters | GMT versus HFT · code generation, Mistral-7B | average score: 58.5 -> 60.0 | Helped |
| Use gradient magnitude to select updates instead of randomly selecting half the parameters | GMT versus HFT · math reasoning, Llama3-8B | GSM8k accuracy: 74.2 -> 82.0 | Helped |
| Vary the mask ratio (ablation) | GMT · single-task code and math benchmarks | benchmark performance | No clear effect |
| Set the mask ratio to an extreme value (ablation) | GMT · general-domain multi-task scenarios | benchmark performance | Hurt |
| Drop salient rather than small-magnitude delta parameters (ablation) | GMT drop-strategy analysis · Mistral-7B math reasoning, GSM8k and MATH | model performance | Hurt |
| Use random dropping rather than dropping small-magnitude delta parameters (ablation) | Drop-strategy analysis · Mistral-7B math reasoning, GSM8k and MATH | model performance | Hurt |
| Drop small-magnitude delta parameters after vanilla fine-tuning (ablation) | One-off Drop · Mistral-7B math reasoning, GSM8k and MATH | model performance | No clear effect |
Surprising findings
- Random masking methods (RMT, HFT) can outperform SFT in the multi-task general domain, suggesting sparsity itself helps.
- A one-off drop of small delta parameters after SFT does not improve performance, unlike in-training masking.
Limitations
- Mask ratio effectiveness diminishes in multi-task general-domain settings at extreme sparsity levels (95%, 99%).
- Gains are modest (around 1 to 3 points) and reported from single runs without variance estimates.
- Training throughput is somewhat lower than vanilla SFT due to the TopK operation.
Experimental setup
Datasets: Magicoder-Evol-Instruct-110K MetaMathQA Tulu V2 UltraFeedback
Benchmarks: HumanEval HumanEval+ MBPP MBPP+ GSM8k MATH MMLU BBH TyDiQA TruthfulQA
Metrics: pass@1 accuracy exact match F1 MC2 training speed (samples/s) FLOPs
Baselines: SFT Drop HFT RMT DPO PAFT
Builds on: Random Mask-Tuning DARE HFT DropBP PAFT
Closest papers by meaning
- Unleashing the Potential of Large Language Models as Prompt Optimizers: Analogical Analysis with Gradient-based Model Optimizers (AAAI 2025 · similarity 0.92)
- Benchmarking Optimizers for Large Language Model Pretraining (NeurIPS 2026 · similarity 0.91)
- Random Masking Finds Winning Tickets for Parameter Efficient Fine-tuning (ICML 2024 · similarity 0.91)
- Optimization Inspired Few-Shot Adaptation for Large Language Models (NeurIPS 2025 · similarity 0.90)
- DELIFT: Data Efficient Language model Instruction Fine-Tuning (ICLR 2025 · similarity 0.90)
- LensLLM: Unveiling Fine-Tuning Dynamics for LLM Selection (ICML 2025 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.