Research map / Learning theory and algorithms / Matrix and tensor methods
Layered-Parameter Perturbation for Zeroth-Order Optimization of Optical Neural Networks
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes layered-parameter perturbation for zeroth-order training of MZI-based optical neural networks, sampling perturbations from a Gaussian with a covariance designed to make output perturbations isotropic.
Problem
Training optical neural networks with fabrication variations cannot use exact backpropagation, so black-box zeroth-order optimization is used; standard isotropic parameter perturbations ignore that MZI parameters in layered modules are interrelated, yielding anisotropic output perturbations.
Approach
Perturb each layered linear module's parameters with N(0, Σu), where Σu is computed in closed form from averaged Fisher information (via Jacobian-vector and vector-Jacobian products) so that output perturbations become as isotropic as possible, minimizing a LogDet divergence cost.
Key results
- ZO-Σu reached 94.70% test accuracy on MNIST with K=64, versus 94.37% for ZO-I and 94.43% for ZO-co.
- On FashionMNIST with K=64, ZO-Σu achieved 81.87% versus 81.51% (ZO-I) and 81.48% (ZO-co).
- The proposed method let truncated Clements(64,48) outperform full Clements(64,64) trained by conventional ZO methods.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use layered-parameter perturbations with Σu instead of identity-covariance perturbations | ZO optimization for ONNs · MNIST, K=16, L=16 | test accuracy: 79.18% ±0.43 -> 79.60% ±0.32 | Helped |
| Use layered-parameter perturbations with Σu instead of coordinate-wise perturbations | ZO optimization for ONNs · MNIST, K=32, L=32 | test accuracy: 92.17% ±0.16 -> 92.29% ±0.21 | Helped |
| Use layered-parameter perturbations with Σu instead of coordinate-wise perturbations | ZO optimization for ONNs · MNIST, K=64, L=64 | test accuracy: 94.43% ±0.15 -> 94.70% ±0.14 | Helped |
| Use layered-parameter perturbations with Σu instead of coordinate-wise perturbations | ZO optimization for ONNs · FashionMNIST, K=16, L=16 | test accuracy: 64.89% ±0.34 -> 65.34% ±0.29 | Helped |
| Use layered-parameter perturbations with Σu instead of coordinate-wise perturbations | ZO optimization for ONNs · FashionMNIST, K=32, L=32 | test accuracy: 77.12% ±0.26 -> 77.43% ±0.18 | Helped |
| Use layered-parameter perturbations with Σu instead of coordinate-wise perturbations | ZO optimization for ONNs · FashionMNIST, K=64, L=64 | test accuracy: 81.48% ±0.18 -> 81.87% ±0.22 | Helped |
Surprising findings
- CMA-ES did not complete a K=64 run within one day, indicating poor scalability.
Limitations
- Validated only in simulation with simulated fabrication errors, not on an actual silicon photonic device.
- Only small feed-forward ONN on MNIST/FashionMNIST; no convolutional or larger models.
- Covariance computed per linear module, not jointly across the whole ONN; extra computational overhead.
Experimental setup
Datasets: MNIST FashionMNIST
Benchmarks: MNIST FashionMNIST
Metrics: test accuracy training loss
Baselines: ZO-I (identity covariance perturbation) ZO-co (coordinate-wise perturbation) CMA-ES backpropagation with error info
Builds on: zeroth-order optimization (Liu et al. 2020) CMA-ES Clements mesh
Closest papers by meaning
- Natural Perturbations for Black-box Training of Neural Networks by Zeroth-Order Optimization (ICML 2025 · similarity 0.86)
- Physics-Constrained Comprehensive Optical Neural Networks (NeurIPS 2024 · similarity 0.85)
- Neural Tangent Knowledge Distillation for Optical Convolutional Networks (NeurIPS 2025 · similarity 0.84)
- $\boldsymbol{\mu}\mathbf{P^2}$: Effective Sharpness Aware Minimization Requires Layerwise Perturbation Scaling (NeurIPS 2024 · similarity 0.82)
- Neural Characteristic Activation Analysis and Geometric Parameterization for ReLU Networks (NeurIPS 2024 · similarity 0.81)
- Improving the Straight-Through Estimator with Zeroth-Order Information (NeurIPS 2025 · similarity 0.81)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.