Research map / Large language models / Instruction and fine-tuning
Low-cost Full Fine-tuning: Learning What to Update for LLMs
Jin Li, Yaming Guo, Shenghao Gao, Xinlong Chen, Zuhao Xu, Ying Sun, Chao Wang, Hui Xiong
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| raise per-step budget from M=32 to M=128 (ablation) | Think-Touch bandit selector (Qwen2.5-1.5B) · Commonsense reasoning avg | accuracy (%): 77.10 -> 81.14 | Improved |
| raise per-step budget from M=32 to M=128 (ablation) | Think-Touch bandit selector (Llama-3.2-3B) · Commonsense reasoning avg | accuracy (%): 81.70 -> 83.83 | Improved |
| use online bandit selector (M=128) instead of LoRA r=64 | Qwen2.5-1.5B · Commonsense reasoning avg | accuracy (%): 80.16 -> 81.14 | Improved |
| use online bandit selector (M=128) instead of LoRA r=64 | Llama-3.2-3B · Commonsense reasoning avg | accuracy (%): 81.83 -> 83.83 | Improved |
| use bandit selector M=32 instead of LoRA r=32 | Qwen2.5-1.5B · Commonsense reasoning avg | accuracy (%): 79.70 -> 77.10 | Hurt |
| use bandit selector (M=128) instead of RandLoRA r=64 | Qwen2.5-1.5B · Commonsense reasoning avg | accuracy (%): 80.83 -> 81.14 | No clear effect |
| use bandit selector (M=128) instead of RandLoRA r=64 | Llama-3.2-3B · Commonsense reasoning avg | accuracy (%): 83.35 -> 83.83 | No clear effect |
| use bandit selector (M=128) instead of DoRA r=64 | Llama-3.2-3B · Commonsense reasoning avg | accuracy (%): 82.74 -> 83.83 | No clear effect |
| use bandit selector instead of uniform group selection (ablation) | Think-Touch · Commonsense reasoning training loss | training loss | Improved |
| update only selected rows (M=128) instead of full fine-tuning | Qwen2.5-1.5B · Commonsense reasoning avg | accuracy (%): 82.49 -> 81.14 | Hurt |
| update only selected rows (M=128) instead of full fine-tuning | Llama-3.2-3B · Commonsense reasoning avg | accuracy (%): 84.50 -> 83.83 | Hurt |
| update selected rows per step instead of full fine-tuning | Qwen2.5-1.5B · wall-clock time | seconds per step: 1.2803s -> 0.1632s | Improved |
Closest papers by meaning
- Data Efficient Adaptation in Large Language Models via Continuous Low-Rank Fine-Tuning (NeurIPS 2025 · similarity 0.94)
- Beyond LoRA vs. Full Fine-Tuning: Gradient-Guided Optimizer Routing for LLM Adaptation (NeurIPS 2026 · similarity 0.92)
- LensLLM: Unveiling Fine-Tuning Dynamics for LLM Selection (ICML 2025 · similarity 0.92)
- Loquetier: A Virtualized Multi-LoRA Framework for Unified LLM Fine-tuning and Serving (NeurIPS 2025 · similarity 0.91)
- Enhancing Zeroth-order Fine-tuning for Language Models with Low-rank Structures (ICLR 2025 · similarity 0.91)
- Linearization Explains Fine-Tuning in Large Language Models (NeurIPS 2025 · similarity 0.91)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
