Research map / Probabilistic and scientific modeling / Bayesian optimization
Natural Perturbations for Black-box Training of Neural Networks by Zeroth-Order Optimization
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes natural perturbations, sampling zeroth-order perturbations from a Gaussian with covariance (λP I + λF F)^-1 that maximizes entropy while regularizing function-space discrepancy via the Fisher information, enabling black-box training of hardware neural networks.
Problem
Training neural networks implemented in hardware (e.g., optical/analog) where backpropagation is inaccurate because internal information is unavailable, so zeroth-order optimization must be used, and standard isotropic or coordinate-wise perturbation sampling is inefficient for deep, correlated para
Approach
Sample ZO perturbations from N(0, Σ) with Σ chosen by maximizing entropy under parameter-space and Fisher-based function-space regularization (analogous to natural gradient); make it practical with block-coordinate partitioning and a black-box, query-based Fisher computation via column-wise Jacobian estimates.
Key results
- ZO-NP improved test accuracy over ZO-I and ZO-co on all five tasks with statistical significance, e.g., FashionMNIST 0.858 vs 0.814 (ZO-co) and CIFAR10 0.625 vs 0.601.
- On Equalization, ZO-NP reached test R2 0.993 vs 0.944 for ZO-I and ZO-co.
- Per-epoch time overhead of ZO-NP was up to about 10%, but memory overhead was substantial (e.g., 2.74 GB vs 0.57 GB on MNIST) unless block size was reduced.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use ZO-NP instead of the strongest baseline ZO-co | ZO training · MNIST | test accuracy: 0.966 ±0.001 -> 0.978 ±0.002 | Helped |
| use ZO-NP instead of the strongest baseline ZO-co | ZO training · FashionMNIST | test accuracy: 0.814 ±0.005 -> 0.858 ±0.004 | Helped |
| use ZO-NP instead of ZO-I or ZO-co | ZO training · Equalization | test R2: 0.944 ±0.007 -> 0.993 ±0.003; 0.944 ±0.008 -> 0.993 ±0.003 | Helped |
| use ZO-NP instead of the strongest baseline ZO-co | ZO training · Copying memory | test accuracy: 0.985 ±0.002 -> 0.993 ±0.003 | Helped |
| use ZO-NP instead of the strongest baseline ZO-co | ZO training · CIFAR10 | test accuracy: 0.601 ±0.004 -> 0.625 ±0.004 | Helped |
| increase perturbation count from Q=1 to Q=100 (ablation) | ZO-NP on FashionMNIST · FashionMNIST | test accuracy: 0.714 -> 0.873 | Helped |
| increase maximum block size Nmax (ablation) | ZO-NP on FashionMNIST · FashionMNIST | test accuracy | Helped |
| use natural perturbations with λF > 0 instead of λF = 0 (ablation) | ZO-NP · MNIST, Equalization, and Copying memory | test accuracy | Helped |
| vary the covariance update frequency Tud (ablation) | ZO-NP · MNIST, Equalization, and Copying memory | test accuracy | No clear effect |
Surprising findings
- Coordinate-wise perturbations (ZO-co) with one-hot vectors were often more effective than isotropic Gaussian perturbations (ZO-I), and ZO-NP with small blocks reduced to the same behavior as ZO-co.
Limitations
- Block-diagonal FIM approximation; memory and query overhead grow with block size, and scaling beyond around one million parameters is a practical limitation.
- Black-box Jacobian computation costs N extra queries, reducing epochs under a fixed query budget, and hurts very small Q settings (Q=1).
- Hyperparameters lambda_F must be tuned per task.
Experimental setup
Datasets: MNIST FashionMNIST CIFAR10 Equalization (synthetic) Copying memory (synthetic)
Benchmarks: MNIST FashionMNIST CIFAR10 Equalization Copying memory
Metrics: test accuracy R2 (coefficient of determination) elapsed time per epoch memory footprint
Baselines: ZO-I (isotropic Gaussian perturbations) ZO-co (coordinate-wise perturbations) CMA-ES
Builds on: natural gradient zeroth-order optimization with Gaussian smoothing Sawada et al. 2025 natural perturbations for optical neural networks Adam
Closest papers by meaning
- Revisiting Zeroth-Order Optimization: Minimum-Variance Two-Point Estimators and Directionally Aligned Perturbations (ICLR 2025 · similarity 0.86)
- Layered-Parameter Perturbation for Zeroth-Order Optimization of Optical Neural Networks (AAAI 2025 · similarity 0.86)
- Min-Max Optimization without Gradients: Convergence and Applications to Black-Box Evasion and Poisoning Attacks (ICML 2020 · similarity 0.85)
- Learning to Learn by Zeroth-Order Oracle (ICLR 2020 · similarity 0.85)
- On the Optimal Construction of Unbiased Gradient Estimators for Zeroth-Order Optimization (NeurIPS 2025 · similarity 0.84)
- Zeroth-Order Optimization Finds Flat Minima (NeurIPS 2025 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.