Research map / Architectures and efficiency / Pruning and efficient tuning

AR1-ZO: Topology-Aware Rank-1 Zeroth-Order Queries for High-Rank LoRA Fine-Tuning

NeurIPS 2026 · Architectures and efficiency › Pruning and efficient tuning · approach: LoRA variants and theory · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes AR1-ZO, which queries one complete rank-1 LoRA atom per step with two-point zeroth-order estimates and rescales the adapter by gamma=alpha*r to keep the finite-difference signal rank-invariant.

Problem

Zeroth-order LoRA fine-tuning of LLMs: full-adapter two-point ZO perturbs r(d_out+d_in) coordinates, and with canonical alpha/r scaling, atomwise queries make the finite-difference signal shrink with rank, causing directional collapse.

Approach

Treat each matched rank-1 LoRA atom (b_k, a_k) as the minimal complete ZO query block (dimension d_out+d_in), cycle through atoms, and replace the joint scaling alpha/r with a topology-aware scale gamma=alpha*r so the active atom's finite-difference signal is rank-invariant.

Key results

• On OPT-2.7B (r=64), AR1-ZO improves over MeZO-LoRA on all six tasks, e.g. SQuAD 77.8 vs 64.6 and CB 69.6 vs 66.1.
• On Qwen3-1.7B (r=64), AR1-ZO improves over MeZO-LoRA by +1.0 to +14.7 points across tasks, e.g. SQuAD 76.7 vs 62.0.
• Naive scaling (ZO-Alt-naive) collapses on SQuAD (F1 about 0.17) while gamma=alpha*r reaches F1 0.770 on Qwen3-1.7B SQuAD ablation.

Surprising findings

• A 10x learning-rate increase on naive-scaled atomwise ZO only partially recovers performance (F1 0.413) and 100x diverges, indicating scale cannot be replaced by learning rate.
• The HIZOO public implementation performed poorly under the matched LoRA-ZO evaluation (SQuAD 0.0 F1 on OPT-2.7B).

Limitations

• Full-adapter stationarity pays an amortized coverage cost across r atoms.
• Experiments limited to OPT (2.7B/13B) and Qwen3 (1.7B/32B) and mostly classification-style or short-answer tasks; free-form generation untested.
• Theoretical rank-invariance relies on active-atom gradient regularity and stable represented-adapter scale assumptions.

Experimental setup

Datasets: BoolQ CB COPA WIC SQuAD DROP

Benchmarks: BoolQ CB COPA WIC SQuAD DROP

Metrics: accuracy F1 finite-difference SNR directional fidelity (cosine)

Baselines: MeZO-LoRA LOZO ZO-Alt-naive HIZOO FT (Adam) MeZO

Builds on: MeZO LoRA LOZO rsLoRA

Closest papers by meaning

• Activated LoRA: Fine-tuned LLMs for Intrinsics (NeurIPS 2025 · similarity 0.87)
• AuroRA: Breaking Low-Rank Bottleneck of LoRA with Nonlinear Mapping (NeurIPS 2025 · similarity 0.87)
• Enhancing Zeroth-order Fine-tuning for Language Models with Low-rank Structures (ICLR 2025 · similarity 0.87)
• Powering Up Zeroth-Order Training via Subspace Gradient Orthogonalization (NeurIPS 2026 · similarity 0.86)
• Beyond Zero Initialization: Investigating the Impact of Non-Zero Initialization on LoRA Fine-Tuning Dynamics (ICML 2025 · similarity 0.86)
• Beyond LoRA vs. Full Fine-Tuning: Gradient-Guided Optimizer Routing for LLM Adaptation (NeurIPS 2026 · similarity 0.86)