Research map / Adversarial robustness and security / Certified and empirical robustness
DeepZero: Scaling Up Zeroth-Order Optimization for Deep Model Training
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes DeepZero, a zeroth-order deep learning framework that scales ZO optimization to training DNNs from scratch via coordinate-wise gradient estimation, ZO-GraSP sparsity, feature reuse and forward parallelization.
Problem
Zeroth-order optimization has been limited to small-scale problems; it is unclear how to train deep neural networks from scratch with function-value-only gradient estimates without large accuracy loss.
Approach
Uses deterministic coordinate-wise gradient estimation (CGE) instead of randomized vector-wise estimation, and induces sparsity via a zeroth-order version of gradient signal preservation (ZO-GraSP) with layer-wise pruning ratios and dynamic random sparsity patterns, plus feature reuse and forward parallelization to speed up training.
Key results
- DeepZero trains ResNet-20 on CIFAR-10 to 86.94% test accuracy, reported as the best gradient-free training result in the literature and approaching FO training.
- On an 8-layer CNN, DeepZero (width 64) reaches 64.1% accuracy versus 58.0% for Align-ada at width 512.
- On ImageNet-10 black-box defense, DeepZero improves certified accuracy over ZO-AE-DS (e.g., 61.80 vs 43.13 at radius 0.5).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Replace randomized vector-wise gradient estimation with coordinate-wise gradient estimation | ZO training of deep CNNs · CIFAR-10 | test accuracy | Helped |
| Use coordinate-wise gradient estimation instead of randomized vector-wise estimation | ZO training of CNNs · CNNs of varying parameter counts | computation time | Helped |
| Add feature reuse | CGE-based ZO training · Deep model training | training time: 2× reduction in training time | Helped |
| Use ZO-GraSP pruning instead of random pruning | ResNet-20 and ResNet-18 sparse models trained on CIFAR-10 · CIFAR-10 | testing accuracy | Helped |
| Use ZO-GraSP pruning instead of FO-GraSP pruning | ResNet-20 and ResNet-18 sparse models trained on CIFAR-10 · CIFAR-10 | testing accuracy | No clear effect |
| Use gradient sparsity rather than directly training a weight-sparse model | DeepZero · CIFAR-10, ResNet-20, sparsity regime 90% to 99% | accuracy | Helped |
| Use SR-guided dynamic sparsity instead of alternating ZO-GraSP and sparse training (M1) or pruning before training (M2) (ablation) | ZO training · Pruning performance across pruning ratios | accuracy | Helped |
| Use DeepZero instead of Pattern Search | ResNet-20 · CIFAR-10, dataset sizes from 100 to 32K | accuracy | Mixed |
| Use DeepZero instead of ZO-AE-DS | Black-box adversarial defense · ImageNet-10, radius r=0.25 | certified accuracy: 52.80 -> 76.61 | Helped |
| Use DeepZero instead of ZO-AE-DS | Black-box adversarial defense · ImageNet-10, radius r=0.5 | certified accuracy: 43.13 -> 61.80 | Helped |
| Use DeepZero instead of ZO-AE-DS | Black-box adversarial defense · ImageNet-10, radius r=0.75 | certified accuracy: 32.73 -> 43.05 | Helped |
| Increase batch size beyond 2048 (ablation) | DeepZero · CIFAR-10, ResNet-20 | performance | Hurt |
Surprising findings
- Coordinate-wise gradient estimation outperforms randomized vector-wise estimation even when the randomized method uses q = d queries.
- Gradient sparsity used inside CGE (training a dense model) outperforms directly training a weight-sparse model at 90-99% sparsity.
Limitations
- High number of model queries inherent to ZO optimization.
- Sparse deep priors may not suit non-deep-learning models.
- Scalability remains a challenge for larger models and datasets; accuracy gap to FO training persists at high sparsity.
Experimental setup
Datasets: CIFAR-10 ImageNet (10-class subset) 2D unsteady wake flow simulations (PhiFlow)
Benchmarks: CIFAR-10 ImageNet (10-class subset) 2D unsteady wake flow PDE error correction
Metrics: test accuracy certified accuracy mean absolute error runtime
Baselines: First-order SGD Pattern Search Align-ada FA DFA FO-DS ZO-AE-DS NON SRC FO-SOL
Builds on: GraSP Smart Ratio Pattern Search Align-ada ZO-AE-DS Solver-in-the-loop
Closest papers by meaning
- Improving the Straight-Through Estimator with Zeroth-Order Information (NeurIPS 2025 · similarity 0.88)
- Refining Adaptive Zeroth-Order Optimization at Ease (ICML 2025 · similarity 0.88)
- Powering Up Zeroth-Order Training via Subspace Gradient Orthogonalization (NeurIPS 2026 · similarity 0.87)
- Revisiting Zeroth-Order Optimization for Memory-Efficient LLM Fine-Tuning: A Benchmark (ICML 2024 · similarity 0.87)
- Trust the Direction, Search the Step: Zero-and-First-Order Methods for LLM Fine-Tuning (NeurIPS 2026 · similarity 0.87)
- Towards Efficient Low-Order Hybrid Optimizer for Language Model Fine-Tuning (AAAI 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.