Research map / Large language models / Instruction and fine-tuning
SFT Doesn’t Always Hurt General Capabilities: Revisiting Domain-Specific Fine-Tuning in LLMs
Jiacheng Lin, Zhongruo Wang, Kun Qian, Tian Wang, Arvind Srinivasan, Hansi Zeng, Ruochen Jiao, Xie Zhou, Jiri Gesi, Dakuo Wang, Yufan Guo, Kai Zhong and 5 more
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use smaller learning rate 1e-6 instead of 5e-6 (ablation) | Standard SFT · General capability avg (MedCalc) | average score: 0.3805 -> 0.6924 | Improved |
| use smaller learning rate 1e-6 instead of 5e-6 (ablation) | Standard SFT · MedCalc | accuracy: 0.5583 -> 0.5339 | No clear effect |
| add TALR at lr 5e-6 | Standard SFT · General capability avg (MedCalc, lr 5e-6) | average score: 0.3805 -> 0.5018 | Improved |
| add TALR at lr 5e-6 | Standard SFT · MedCalc domain (lr 5e-6) | accuracy: 0.5583 -> 0.5417 | Hurt |
| add TALR at lr 1e-6 | Standard SFT · General capability avg (MedCalc, lr 1e-6) | average score: 0.6924 -> 0.7169 | Improved |
| use LoRA instead of full fine-tuning | Standard SFT · MedCalc domain (lr 1e-6) | accuracy: 0.5339 -> 0.1813 | Hurt |
| add Wise-FT model averaging | Standard SFT · MedCalc domain (lr 1e-6) | accuracy: 0.5339 -> 0.1983 | Hurt |
| add KL regularization (coefficient 0.001) (ablation) | Standard SFT · MedCalc (Qwen2.5-3B-Instruct) | general performance | No clear effect |
| label-only supervision (no CoT) on ESCI (ablation) | SFT · ESCI | trade-off across learning rates | Improved |
| lower learning rate to 5e-7 on MetaMathQA (ablation) | SFT DeepSeek-Coder-7B · GSM8K | domain accuracy | Hurt |
Closest papers by meaning
- Data Mixing Optimization for Supervised Fine-Tuning of Large Language Models (ICML 2025 · similarity 0.93)
- FisherSFT: Data-Efficient Supervised Fine-Tuning of Language Models Using Information Gain (ICML 2025 · similarity 0.92)
- Data Efficient Adaptation in Large Language Models via Continuous Low-Rank Fine-Tuning (NeurIPS 2025 · similarity 0.92)
- Synergy over Discrepancy: A Partition-Based Approach to Multi-Domain LLM Fine-Tuning (NeurIPS 2025 · similarity 0.91)
- Two-stage LLM Fine-tuning with Less Specialization and More Generalization (ICLR 2024 · similarity 0.91)
- Unveiling the Secret Recipe: A Guide For Supervised Fine-Tuning Small LLMs (ICLR 2025 · similarity 0.91)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
