Research map / Large language models / Instruction and fine-tuning
New Hybrid Fine-Tuning Paradigm for LLMs: Algorithm Design and Convergence Analysis Framework
Shaocong Ma, Peiran Yu, Heng Huang
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| update the base LLM with zeroth-order steps jointly with the first-order PEFT module | FO PEFT (Prompt, Prefix, LoRA) · 6 NLP tasks x 3 models | accuracy: better in 41 of 54 pairs; 17 of 18 aggregate | Improved |
| add hybrid ZO update to LoRA | FO-LoRA · SST-2 (Llama-2-7b) | accuracy: 94.6 -> 93.4 | Hurt |
| add hybrid ZO update to Prompt tuning | FO-Prompt · RTE (Vicuna-7b-v1.5) | accuracy: 82.3 -> 70.1 | Hurt |
| add hybrid ZO update to Prompt tuning | FO-Prompt · WSC (Llama-2-7b) | accuracy: 36.5 -> 61.5 | Improved |
| use hybrid fine-tuning with LoRA on RTE instead of FO-LoRA | FO-LoRA · RTE (OPT-1.3b) | accuracy: 54.8 -> 61.0 | Improved |
| compare Hybrid-LoRA (SGD) with FO-LoRA using Adam | Hybrid-LoRA · 6 tasks (OPT-1.3b) | accuracy: RTE 58.6 -> 61.0; COPA 66.0 -> 78.0; WiC 64.1 -> 55.5; WinoG 60.1 -> 58.3 | Mixed |
| use hybrid ZO-SGD instead of FO full fine-tuning | full fine-tuning · Llama-2-7B SST-2 | peak GPU memory: 54 GB -> 46 GB | Improved |
| hybrid fine-tuning adds ZO update on LLM to FO prompt tuning | FO-SGD (Prompt) · Llama-2-7B SST-2 | peak GPU memory: 46 GB -> 46 GB | No clear effect |
| use hybrid fine-tuning instead of baselines | OPT-1.3B prompt tuning · SST-2 | steps to 90% accuracy: at least 12,500 -> around 2,500 | Improved |
| use distinct learning rates for LLM (1e-6) and prompt module (1e-3) rather than one shared rate (ablation) | hybrid fine-tuning · SST-2 (OPT-1.3b) | training loss | Improved |
| Hybrid vs ZO and FO full fine-tuning | full fine-tuning · 18 model-task combinations | accuracy: 13/18 combinations | Improved |
Closest papers by meaning
- Linearization Explains Fine-Tuning in Large Language Models (NeurIPS 2025 · similarity 0.92)
- When Scaling Meets LLM Finetuning: The Effect of Data, Model and Finetuning Method (ICLR 2024 · similarity 0.91)
- Data Efficient Adaptation in Large Language Models via Continuous Low-Rank Fine-Tuning (NeurIPS 2025 · similarity 0.91)
- LensLLM: Unveiling Fine-Tuning Dynamics for LLM Selection (ICML 2025 · similarity 0.91)
- Revisiting Zeroth-Order Optimization for Memory-Efficient LLM Fine-Tuning: A Benchmark (ICML 2024 · similarity 0.91)
- Data Mixing Optimization for Supervised Fine-Tuning of Large Language Models (ICML 2025 · similarity 0.91)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
