Research map / Large language models / Instruction and fine-tuning
Bilevel ZOFO: Efficient LLM Fine-Tuning and Meta-Training
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Bilevel-ZOFO, which runs FO-PEFT (inner, lower-level) and ZO full-backbone updates (outer) under a penalized bilevel/minimax formulation, to get faster and prompt-robust LLM fine-tuning plus a meta-learning variant.
Problem
Full fine-tuning of LLMs with first-order optimizers is memory-heavy, PEFT often underperforms full fine-tuning, and zeroth-order (MeZO) full fine-tuning converges slowly and is highly sensitive to prompt choice.
Approach
Treat PEFT parameters as the solution of an inner (lower-level) problem updated with first-order gradients, and update the full backbone at the outer level with SPSA zeroth-order gradients, using a penalty-based minimax reformulation of the bilevel problem with convergence guarantees.
Key results
- On OPT-1.3B single-task, Bilevel-ZOFO LoRA averages 0.7410 across 9 tasks vs 0.6748 for MeZO LoRA and 0.7330 for FO LoRA.
- On Llama2-7B, Bilevel-ZOFO averages 0.8394 across four tasks vs 0.8047 for MeZO full fine-tuning and 0.8356 for FO PEFT average.
- Prompt sensitivity: removing the prompt drops MeZO by 38.6 points on SST-2 but Bilevel-ZOFO by only 0.4 points.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use Bilevel-ZOFO with LoRA rather than FO LoRA | OPT-1.3B single-task fine-tuning · OPT-1.3B single-task average | accuracy/F1: 0.7330 -> 0.7410 | Helped |
| use Bilevel-ZOFO rather than MeZO | OPT-1.3B single-task fine-tuning · OPT-1.3B single-task average | accuracy/F1: 0.6529 -> 0.7158 | Helped |
| use Bilevel-ZOFO rather than FO PEFT | Llama2-7B single-task fine-tuning · Llama2-7B single-task average | accuracy/F1: 0.8356 -> 0.8394 | Helped |
| remove the task prompt | Bilevel-ZOFO on SST-2 · SST-2 | accuracy: 93.3 -> 92.9; -0.4 | No clear effect |
| remove the task prompt | MeZO on SST-2 · SST-2 | accuracy: 89.6 -> 51.9; -38.6 | Hurt |
| replace exact first-order PEFT gradients with zeroth-order updates (ablation) | Bilevel-ZOFO · BoolQ, CB, COPA, SST-2 | accuracy: BoolQ 0.6280 vs 0.7367; CB 0.6092 vs 0.7679; COPA 0.7146 vs 0.7633; SST2 0.7633 vs 0.8335 | Hurt |
| replace bilevel optimization with a two-stage FO-then-ZO pipeline (ablation) | Bilevel-ZOFO · BoolQ | accuracy: 0.7060 vs 0.7367 | Hurt |
| jointly optimize backbone and PEFT parameters instead of using the bilevel structure (ablation) | Bilevel-ZOFO · BoolQ | accuracy: 0.7209 vs 0.7367 | Hurt |
| vary the regularization parameter λ (ablation) | Bilevel-ZOFO · single-task experiments | accuracy | Mixed |
| reduce the number of upper-level updates (ablation) | Bilevel-ZOFO · single-task experiments | accuracy | Hurt |
| vary the lower-level/upper-level data split (ablation) | Bilevel-ZOFO · single-task experiments | accuracy | Mixed |
| LoRA-tune the meta-trained model for 10 additional iterations | Bilevel-ZOFO meta-learning · GPT2-Large meta-learning tasks | Macro-averaged F1: class → class 34.5 -> 47.1; non_class → class 34.3 -> 42.4; qa → qa 41.8 -> 43.5 (1.3); non_qa → qa 40.4 -> 41.9 | Helped |
Surprising findings
- Bilevel-ZOFO's average on Llama2-7B (0.8394) is only marginally above FO PEFT average (0.8356), and several FO PEFT LoRA results on some tasks are higher than the ZO-based method's.
- Jointly optimizing backbone and PEFT parameters with the same objective gives only modest gains over MeZO and FO PEFT, while the bilevel structure gives larger improvements.
Limitations
- Theory assumes strong convexity of the lower-level objective, which the experiments do not strictly satisfy.
- Large-scale LLMs were not tested due to resource constraints.
- Uses classic MeZO as the upper-level ZO method; other ZO variants not explored.
Experimental setup
Datasets: SST-2 BoolQ CB COPA ReCoRD RTE WIC WinoGrande WSC SQuAD CrossFit UnifiedQA
Benchmarks: SST-2 BoolQ CB COPA ReCoRD RTE WIC WinoGrande WSC SQuAD
Metrics: accuracy F1 macro-averaged F1 peak memory step duration
Baselines: MeZO FO PEFT (LoRA, prefix, prompt tuning) Bilevel-ZOZO Two-Stage Pipeline MetaICL ICL
Builds on: MeZO penalty-based bilevel optimization LoRA prefix tuning MetaICL
Closest papers by meaning
- Revisiting Zeroth-Order Optimization for Memory-Efficient LLM Fine-Tuning: A Benchmark (ICML 2024 · similarity 0.92)
- Enhancing Zeroth-order Fine-tuning for Language Models with Low-rank Structures (ICLR 2025 · similarity 0.92)
- Harmony in Divergence: Towards Fast, Accurate, and Memory-efficient Zeroth-order LLM Fine-tuning (NeurIPS 2025 · similarity 0.91)
- Variance-reduced Zeroth-Order Methods for Fine-Tuning Language Models (ICML 2024 · similarity 0.90)
- Towards Efficient Low-Order Hybrid Optimizer for Language Model Fine-Tuning (AAAI 2025 · similarity 0.90)
- Dominant-Layer ZO: A Single Layer Dominates Zeroth-Order Fine-Tuning of LLMs (NeurIPS 2026 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.