Research map / Architectures and efficiency / Pruning and efficient tuning
ENGINE: Endogenous Variational MultiScale Optimization for Zeroth-Order LLM Fine-Tuning
Zhuoli Ouyang, Changxi Chi, Siyuan Li, Tailin Wu
Read the paperCheck your idea against 137,217 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes ENGINE, an endogenous variational multiscale zeroth-order optimizer for memory-efficient LLM fine-tuning that aims to reduce gradient variance without discarding high-dimensional information.
Problem
Zeroth-order LLM fine-tuning suffers from exploding gradient variance and high cost in the massive parameter space, while confining updates to low-dimensional subspaces reduces variance but loses critical high-dimensional information.
Approach
Combines curvature-aware zeroth-order updates with a variational multiscale view so that updates are organized across scales rather than confined to one low-dimensional subspace; details beyond the abstract are not available.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add VMS-EO subgrid correction to the V-cycle (ablation) | ENGINE · GLUE avg, GPT-2-XL | accuracy (%): 70.8 -> 73.4 | Improved |
| add VMS-EO subgrid correction to the V-cycle (ablation) | ENGINE · SuperGLUE avg, OPT-13B | accuracy (%): 64.0 -> 66.4 | Improved |
| add VMS-EO subgrid correction to the V-cycle (ablation) | ENGINE · SuperGLUE avg, LLaMA-3-8B | accuracy (%): 63.2 -> 64.7 | Improved |
| V-cycle alone (coarse/fine alternation) without correction versus LOREN (ablation) | ENGINE w/o VMS-EO · 16 GLUE/SuperGLUE settings | accuracy (%): GPT-2-XL avg 71.4 -> 70.8; OPT-2.7B avg 70.8 -> 71.8 | Mixed |
| replace LOREN with ENGINE (V-cycle plus correction) | ENGINE · GLUE avg, GPT-2-XL | accuracy (%): 71.4 -> 73.4 | Improved |
| replace LOREN with ENGINE (V-cycle plus correction) | ENGINE · SuperGLUE avg, OPT-13B | accuracy (%): 63.4 -> 66.4 | Improved |
| replace LOREN with ENGINE (V-cycle plus correction) | ENGINE · SuperGLUE CB, OPT-13B | accuracy (%): 73.7 -> 79.1 | Improved |
| replace LOREN with ENGINE | ENGINE · LLaMA-3-8B CB | forward passes to LOREN's best accuracy: 2.61x speed-up | Improved |
| replace LOREN with ENGINE | ENGINE · peak GPU memory | peak memory | No clear effect |
Experimental setup
Builds on: MeZO
Closest papers by meaning
- Second-Order Fine-Tuning without Pain for LLMs: A Hessian Informed Zeroth-Order Optimizer (ICLR 2025 · similarity 0.91)
- Harmony in Divergence: Towards Fast, Accurate, and Memory-efficient Zeroth-order LLM Fine-tuning (NeurIPS 2025 · similarity 0.90)
- Enhancing Zeroth-order Fine-tuning for Language Models with Low-rank Structures (ICLR 2025 · similarity 0.90)
- Revisiting Zeroth-Order Optimization for Memory-Efficient LLM Fine-Tuning: A Benchmark (ICML 2024 · similarity 0.89)
- S$^{2}$FT: Efficient, Scalable and Generalizable LLM Fine-tuning by Structured Sparsity (NeurIPS 2024 · similarity 0.89)
- Sparse MeZO: Less Parameters for Better Performance in Zeroth-Order LLM Fine-Tuning (NeurIPS 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
