Research map / Architectures and efficiency / Pruning and efficient tuning

Powering Up Zeroth-Order Training via Subspace Gradient Orthogonalization

NeurIPS 2026 · Architectures and efficiency › Pruning and efficient tuning · approach: Low-rank compression and merging · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes ZO-Muon, which applies Muon-style gradient orthogonalization inside a randomly projected low-rank subspace of zeroth-order gradient estimates, improving accuracy and query/runtime efficiency for LLM and ViT fine-tuning.

Problem

Zeroth-order fine-tuning of large models suffers from high gradient-estimation variance that grows with dimension, creating a poor accuracy-query efficiency trade-off; multi-query averaging yields diminishing returns.

Approach

Estimate the zeroth-order gradient in a random column-orthonormal subspace (Subspace RGE), apply Muon's msign gradient orthogonalization to the low-dimensional estimate, and lift the result back to full space, so orthogonalization acts as a low-rank spectral filter on noisy ZO gradients.

Key results

• ZO-Muon requires only 24.7% of the queries of MeZO to reach the same SST-2 performance for LLM fine-tuning.
• On ViT-L CIFAR-100 fine-tuning, ZO-Muon outperforms MeZO by 25.1% in accuracy and the second-best ZO method Subspace-MeZO by 7.7%.
• On OPT-13B SST-2 with a 40k query budget, ZO-Muon reaches 92.5% accuracy vs. 91.4% for MeZO while requiring 70.5% of MeZO's runtime.

Surprising findings

• Applying Muon orthogonalization directly to full-space ZO gradients (ZO-Muon-V0, JAGUAR-Muon) fails to beat MeZO, while the same operation inside a random subspace helps.
• Subspace-MeZO exhibits a multi-query paradox (N_q>1 worse than N_q=1), which ZO-Muon avoids.

Limitations

• Focuses on parameter-efficient fine-tuning; extension to full pre-training is left open.
• Hyperparameter grids (rank, N_q, steps) are tuned per setting; SubZero underperforms on ViT despite tuning.
• Several competing methods (FZOO, MUZO) were not compared because no open-source implementations were found.

Experimental setup

Datasets: SST-2 RTE CB BoolQ WiC SQuAD CIFAR-10 CIFAR-100

Benchmarks: SST-2 RTE CB BoolQ WiC SQuAD CIFAR-10 CIFAR-100

Metrics: accuracy F1 runtime number of queries GPU memory

Baselines: MeZO SparseMeZO HiZOO LOZO SubZero Subspace-MeZO Adam LoRA JAGUAR-Muon

Builds on: MeZO Muon LOZO SubZero Newton-Schulz iteration

Closest papers by meaning

• Refining Adaptive Zeroth-Order Optimization at Ease (ICML 2025 · similarity 0.88)
• Enhancing Zeroth-order Fine-tuning for Language Models with Low-rank Structures (ICLR 2025 · similarity 0.88)
• DeepZero: Scaling Up Zeroth-Order Optimization for Deep Model Training (ICLR 2024 · similarity 0.87)
• Sparse MeZO: Less Parameters for Better Performance in Zeroth-Order LLM Fine-Tuning (NeurIPS 2025 · similarity 0.87)
• ZO-AdaMU Optimizer: Adapting Perturbation by the Momentum and Uncertainty in Zeroth-Order Optimization (AAAI 2024 · similarity 0.87)
• Harmony in Divergence: Towards Fast, Accurate, and Memory-efficient Zeroth-order LLM Fine-tuning (NeurIPS 2025 · similarity 0.87)