Research map / Optimization / Optimizers
Accelerating PDE-Constrained Optimization by the Derivative of Neural Operators
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes an optimization-oriented training scheme for reference neural operators (RNO) plus a Virtual-Fourier layer and a hybrid solver-in-the-loop optimizer to make neural-operator-based gradient optimization for PDE-constrained problems accurate and stable.
Problem
Neural operator surrogates for gradient-based PDE-constrained optimization suffer from data inefficiency (optimization data is highly correlated and poorly sampled) and instability (errors in predicted solutions and derivatives accumulate and derail optimization).
Approach
Train neural operators on full optimization trajectories using nearest-neighbor reference-query pairs (RNO), supervise sensitivities with a sensitivity loss, replace Fourier/transformer layers with a Virtual-Fourier layer that has simpler derivatives, and periodically recalibrate optimization with numerical solver calls (hybrid optimization) plus gradient smoothing.
Key results
- R-VF achieves the best error on most components across four datasets; e.g., Microreactor2D sensitivity error 1.70e-1 vs 2.02e-1 for second best R-PA.
- Hybrid R-VF optimization converges stably on Microreactor2D while R-VF without solver feedback derails.
- R-VF on Microreactor2D runs at 0.53 s/iter on GPU vs 3.20 s/iter for numerical CPU solver; on Drone3D 0.62 vs 49.40 s/iter.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use the VF layer instead of physics attention in the RNO framework | RNO · Microreactor2D | relative-l2 error (p): 1.38e-2 -> 1.33e-2 | Helped |
| Use the VF layer instead of physics attention in the RNO framework | RNO · Fuelcell2D | relative-l2 error (p): 1.34e-2 -> 1.30e-2 | Helped |
| Use the VF layer instead of linear attention in the RNO framework | RNO · Inductor2D | relative-l2 error (B_r): 9.65e-3 -> 8.66e-3 | Helped |
| Use the VF layer instead of physics attention in the RNO framework | RNO · Drone3D | relative-l2 error (u_1): 9.60e-2 -> 7.64e-2 | Helped |
| Use the VF layer instead of physics attention in the RNO framework | RNO · Microreactor2D | sensitivity error (L_s): 2.02e-1 -> 1.70e-1 | Helped |
| Use the VF layer instead of physics attention in the RNO framework | RNO · Fuelcell2D | sensitivity error (L_1): 4.62e-1 -> 4.44e-1 | Helped |
| Use the VF layer instead of linear attention in the RNO framework | RNO · Inductor2D | sensitivity error (L_r): 3.12e-1 -> 3.00e-1 | Helped |
| Use the VF layer instead of physics attention in the RNO framework | RNO · Drone3D | sensitivity error (L_s): 5.66e-1 -> 5.05e-1 | Helped |
| Remove reference inputs (R-VF w/o ref) (ablation) | R-VF · Microreactor2D | relative-l2 error (p): 1.33e-2 -> 5.56e-2 | Hurt |
| Remove reference inputs (R-VF w/o ref) (ablation) | R-VF · Fuelcell2D | relative-l2 error (p): 1.30e-2 -> 1.05e-1 | Hurt |
| Remove reference inputs (R-VF w/o ref) (ablation) | R-VF · Inductor2D | relative-l2 error (B_r): 8.66e-3 -> 3.93e-1 | Hurt |
| Remove reference inputs (R-VF w/o ref) (ablation) | R-VF · Drone3D | relative-l2 error (u_1): 7.64e-2 -> 4.42e-1 | Hurt |
Surprising findings
- Vanilla neural operators trained on optimization trajectories perform poorly, while the RNO training procedure even without reference (reference dropout 1) outperforms all traditional neural operators.
Limitations
- Hybrid optimization still requires periodic calls to the numerical solver, and may converge to suboptimal solutions due to derivative and objective discrepancies.
- Method relies on smooth dependence of solutions on design parameters; large topology changes are not supported.
- Objective approximations and Lagrange-multiplier constraint handling introduce sensitivity discrepancies.
Experimental setup
Datasets: Microreactor2D Fuelcell2D Inductor2D Drone3D
Benchmarks: Microreactor2D Fuelcell2D Inductor2D Drone3D
Metrics: relative l2 error sensitivity error objective value during optimization wall-clock time per iteration
Baselines: GNOT linear attention (LA) Transolver physics attention (PA) R-LA R-PA numerical adjoint optimizer (COMSOL)
Builds on: Reference neural operators (RNO) Transolver GNOT Fourier neural operator Sobolev training
Closest papers by meaning
- HEAP: Hyper Extended A-PDHG Operator for Constrained High-dim PDEs (ICML 2025 · similarity 0.88)
- Boosting Generalization in Parametric PDE Neural Solvers through Adaptive Conditioning (NeurIPS 2024 · similarity 0.87)
- Derivative-Informed Training of Neural Operators On-the-Fly via Sketched Tangent Consistency (NeurIPS 2026 · similarity 0.87)
- Active Learning for Neural PDE Solvers (ICLR 2025 · similarity 0.86)
- Derivative-enhanced Deep Operator Network (NeurIPS 2024 · similarity 0.86)
- Global convergence of adjoint-optimized neural PDEs (NeurIPS 2026 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.