Research map / Optimization / Optimizers
PseuZO: Pseudo-Zeroth-Order Algorithm for Training Deep Neural Networks
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's abstract only. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes PseuZO, a pseudo-zeroth-order training algorithm for deep neural networks aimed at reducing the dimension-dependent variance of existing zeroth-order gradient estimators.
Problem
Existing zeroth-order (ZO) gradient estimators for training and fine-tuning large models have dimension-dependent variance scaling (Theta(d)), which slows convergence and limits their use in memory-efficient LLM fine-tuning.
Approach
Introduces a pseudo-zeroth-order gradient estimation scheme for deep neural network training that avoids the dimension-dependent variance of standard ZO estimators; full details are not available in the provided text.
Closest papers by meaning
- DeepZero: Scaling Up Zeroth-Order Optimization for Deep Model Training (ICLR 2024 · similarity 0.89)
- Sparse MeZO: Less Parameters for Better Performance in Zeroth-Order LLM Fine-Tuning (NeurIPS 2025 · similarity 0.88)
- Improving the Straight-Through Estimator with Zeroth-Order Information (NeurIPS 2025 · similarity 0.88)
- Variance-reduced Zeroth-Order Methods for Fine-Tuning Language Models (ICML 2024 · similarity 0.88)
- Harmony in Divergence: Towards Fast, Accurate, and Memory-efficient Zeroth-order LLM Fine-tuning (NeurIPS 2025 · similarity 0.87)
- Revisiting Zeroth-Order Optimization for Memory-Efficient LLM Fine-Tuning: A Benchmark (ICML 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.