AI Paper Atlas

Research map / Architectures and efficiency / Pruning and efficient tuning

ENGINE: Endogenous Variational MultiScale Optimization for Zeroth-Order LLM Fine-Tuning

Zhuoli Ouyang, Changxi Chi, Siyuan Li, Tailin Wu

NeurIPS 2026 · Architectures and efficiency › Pruning and efficient tuning · approach: Adapter-based parameter-efficient tuning · 0 citations (OpenAlex)

Read the paperCheck your idea against 137,217 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes ENGINE, an endogenous variational multiscale zeroth-order optimizer for memory-efficient LLM fine-tuning that aims to reduce gradient variance without discarding high-dimensional information.

Problem

Zeroth-order LLM fine-tuning suffers from exploding gradient variance and high cost in the massive parameter space, while confining updates to low-dimensional subspaces reduces variance but loses critical high-dimensional information.

Approach

Combines curvature-aware zeroth-order updates with a variational multiscale view so that updates are organized across scales rather than confined to one low-dimensional subspace; details beyond the abstract are not available.

Measured effects reported in the paper

What was changedWhereMeasuredEffect
add VMS-EO subgrid correction to the V-cycle (ablation)ENGINE · GLUE avg, GPT-2-XLaccuracy (%): 70.8 -> 73.4Improved
add VMS-EO subgrid correction to the V-cycle (ablation)ENGINE · SuperGLUE avg, OPT-13Baccuracy (%): 64.0 -> 66.4Improved
add VMS-EO subgrid correction to the V-cycle (ablation)ENGINE · SuperGLUE avg, LLaMA-3-8Baccuracy (%): 63.2 -> 64.7Improved
V-cycle alone (coarse/fine alternation) without correction versus LOREN (ablation)ENGINE w/o VMS-EO · 16 GLUE/SuperGLUE settingsaccuracy (%): GPT-2-XL avg 71.4 -> 70.8; OPT-2.7B avg 70.8 -> 71.8Mixed
replace LOREN with ENGINE (V-cycle plus correction)ENGINE · GLUE avg, GPT-2-XLaccuracy (%): 71.4 -> 73.4Improved
replace LOREN with ENGINE (V-cycle plus correction)ENGINE · SuperGLUE avg, OPT-13Baccuracy (%): 63.4 -> 66.4Improved
replace LOREN with ENGINE (V-cycle plus correction)ENGINE · SuperGLUE CB, OPT-13Baccuracy (%): 73.7 -> 79.1Improved
replace LOREN with ENGINEENGINE · LLaMA-3-8B CBforward passes to LOREN's best accuracy: 2.61x speed-upImproved
replace LOREN with ENGINEENGINE · peak GPU memorypeak memoryNo clear effect

Experimental setup

Builds on: MeZO

Closest papers by meaning

Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.

Open scime draft

Get the next scime Dispatch and a monthly field update

Free. We send a confirmation link first; every email has a one-click unsubscribe. Privacy