AI Paper Atlas

Research map / Large language models / Instruction and fine-tuning

New Hybrid Fine-Tuning Paradigm for LLMs: Algorithm Design and Convergence Analysis Framework

Shaocong Ma, Peiran Yu, Heng Huang

ICLR 2026 · Large language models › Instruction and fine-tuning · approach: Fine-tuning generalization and delta compression · 1 citations (OpenAlex)

Read the paperCheck your idea against 137,217 papers

AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Measured effects reported in the paper

What was changedWhereMeasuredEffect
update the base LLM with zeroth-order steps jointly with the first-order PEFT moduleFO PEFT (Prompt, Prefix, LoRA) · 6 NLP tasks x 3 modelsaccuracy: better in 41 of 54 pairs; 17 of 18 aggregateImproved
add hybrid ZO update to LoRAFO-LoRA · SST-2 (Llama-2-7b)accuracy: 94.6 -> 93.4Hurt
add hybrid ZO update to Prompt tuningFO-Prompt · RTE (Vicuna-7b-v1.5)accuracy: 82.3 -> 70.1Hurt
add hybrid ZO update to Prompt tuningFO-Prompt · WSC (Llama-2-7b)accuracy: 36.5 -> 61.5Improved
use hybrid fine-tuning with LoRA on RTE instead of FO-LoRAFO-LoRA · RTE (OPT-1.3b)accuracy: 54.8 -> 61.0Improved
compare Hybrid-LoRA (SGD) with FO-LoRA using AdamHybrid-LoRA · 6 tasks (OPT-1.3b)accuracy: RTE 58.6 -> 61.0; COPA 66.0 -> 78.0; WiC 64.1 -> 55.5; WinoG 60.1 -> 58.3Mixed
use hybrid ZO-SGD instead of FO full fine-tuningfull fine-tuning · Llama-2-7B SST-2peak GPU memory: 54 GB -> 46 GBImproved
hybrid fine-tuning adds ZO update on LLM to FO prompt tuningFO-SGD (Prompt) · Llama-2-7B SST-2peak GPU memory: 46 GB -> 46 GBNo clear effect
use hybrid fine-tuning instead of baselinesOPT-1.3B prompt tuning · SST-2steps to 90% accuracy: at least 12,500 -> around 2,500Improved
use distinct learning rates for LLM (1e-6) and prompt module (1e-3) rather than one shared rate (ablation)hybrid fine-tuning · SST-2 (OPT-1.3b)training lossImproved
Hybrid vs ZO and FO full fine-tuningfull fine-tuning · 18 model-task combinationsaccuracy: 13/18 combinationsImproved

Closest papers by meaning

Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.

Open scime draft

Get the next scime Dispatch and a monthly field update

Free. We send a confirmation link first; every email has a one-click unsubscribe. Privacy