Research map / Architectures and efficiency / Pruning and efficient tuning
Towards Efficient Low-Order Hybrid Optimizer for Language Model Fine-Tuning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes LoHO, a low-order hybrid optimizer that combines MeZO (zeroth-order) with first-order optimizers at the layer level (shallow layers ZO, deep layers FO) and weight level (random subset of each layer FO), with gradient-norm-based learning rate customization, to improve MeZO accuracy and convergence within a fixed memory budget.
Problem
Fine-tuning large language models with first-order optimizers requires high memory, while the memory-efficient zeroth-order optimizer MeZO suffers accuracy drops and slow convergence.
Approach
Hybridize ZO and FO optimization: use FO on deep layers (avoiding caching/backprop through shallow layers) and ZO on shallow layers (inter-layer), or split each layer's weights randomly between FO and ZO updates with sparse storage (intra-layer), and scale ZO vs FO learning rates by gradient Frobenius norm differences, using leftover GPU memory to improve MeZO.
Key results
- On RoBERTa-large, LoHO-Adam_inter averages 86.7% vs 84.4% for MeZO-Adam and 83.3% for MeZO across SST-2, RTE, MNLI, SNLI.
- On OPT-13B, LoHO-Adam_inter averages 68.9% vs 64.6% for MeZO across RTE, CB, BoolQ, WIC, MultiRC, running on a single 80GB A800.
- LoHO-Adam_inter on RoBERTa-large reduces memory by over 50% versus full FT with Adam (20.5GB vs 45.8GB), with accuracy close to Full FT (Adam) at 87.2% average.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use inter-layer hybrid optimization instead of MeZO alone | OPT-13B fine-tuning · RTE, CB, BoolQ, WIC, and MultiRC | average accuracy: +2.9% (LoHO-SGDinter) and +4.3% (LoHO-Adaminter) | Helped |
| Use inter-layer hybrid optimization instead of MeZO alone | OPT-30B fine-tuning · RTE, BoolQ, and WIC | average accuracy: 63.3 -> 65.7; 63.3 -> 66.8 | Helped |
| Use inter-layer hybrid optimization with Adam rather than SGD | LoHO fine-tuning · RoBERTa-large, SST-2, RTE, MNLI, and SNLI | average accuracy: 85.2 -> 86.7 | Helped |
| Use hybrid optimization rather than only the FO optimizer (ablation) | Inter-layer LoHO · RoBERTa-large, MNLI and RTE | accuracy | Helped |
| Set separate learning rates for the ZO and FO optimizers rather than equal learning rates (ablation) | Inter-layer LoHO · RoBERTa-large, MNLI and RTE | accuracy | Helped |
| Use random selection instead of absolute-min or absolute-max parameter selection (ablation) | Intra-layer LoHO · RoBERTa-large, MNLI and RTE | accuracy | No clear effect |
| Increase the ratio of FO-optimized parameters (ablation) | Intra-layer LoHO · RoBERTa-large, MNLI | accuracy: 78.3, 79.9, 81.2, 81.7, 81.9, 82.2, 82.6, 83.2 | Helped |
| Use 60% rather than 100% FO-optimized parameters (ablation) | Intra-layer LoHO · RoBERTa-large, MNLI | accuracy: 82.6; 83.2 | No clear effect |
| Place FO-optimized layers in deeper layers rather than shallow layers (ablation) | Inter-layer LoHO-Adam · RoBERTa-large, MNLI | accuracy | Helped |
| Increase the number of FO-optimized deep layers (ablation) | Inter-layer LoHO-Adam · RoBERTa-large, MNLI | accuracy | Helped |
| Use LoHO instead of MeZO | RoBERTa-large fine-tuning · MNLI and RTE | convergence rate | Helped |
| Use LoHO-Adaminter rather than LoHO-SGDinter | RoBERTa-large fine-tuning · MNLI and RTE | convergence rate | Helped |
Surprising findings
- The Frobenius norm of MeZO gradients is much larger than that of SGD gradients, so the ZO and FO learning rates need to differ by orders of magnitude; using equal learning rates performed poorly.
- Random selection of FO-updated parameters within a layer performs comparably to absolute-min and absolute-max heuristic selection.
Limitations
- Number of FO layers is set to the maximum fitting a single GPU rather than tuned, and accuracy on long-sequence tasks drops because fewer FO layers fit in memory.
- Performance on OPT-30B is lower than on OPT-13B because fewer FO layers fit within the memory budget.
- Evaluated only on classification-style tasks with small training sets (512 per class or 1,000 examples).
Experimental setup
Datasets: SST-2 RTE MNLI SNLI BoolQ CB MultiRC WIC
Benchmarks: SST-2 RTE MNLI SNLI BoolQ CB MultiRC WIC
Metrics: accuracy peak GPU memory (GB) training loss vs steps and time
Baselines: Zero-shot ICL MeZO MeZO-Adam Full FT (SGD) Full FT (Adam)
Builds on: MeZO SPSA Adam AdamW
Closest papers by meaning
- Revisiting Zeroth-Order Optimization for Memory-Efficient LLM Fine-Tuning: A Benchmark (ICML 2024 · similarity 0.93)
- Harmony in Divergence: Towards Fast, Accurate, and Memory-efficient Zeroth-order LLM Fine-tuning (NeurIPS 2025 · similarity 0.93)
- Enhancing Zeroth-order Fine-tuning for Language Models with Low-rank Structures (ICLR 2025 · similarity 0.92)
- Variance-reduced Zeroth-Order Methods for Fine-Tuning Language Models (ICML 2024 · similarity 0.91)
- Sparse MeZO: Less Parameters for Better Performance in Zeroth-Order LLM Fine-Tuning (NeurIPS 2025 · similarity 0.91)
- Bilevel ZOFO: Efficient LLM Fine-Tuning and Meta-Training (NeurIPS 2025 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.