Research map / Architectures and efficiency / Attention and transformers
Improved Operator Learning by Orthogonal Attention
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Orthogonal Neural Operator (ONO), whose core orthogonal attention parameterizes orthonormal neural eigenfunctions via an EMA-estimated covariance and Cholesky whitening to act as a regularized kernel integral operator for PDE surrogate learning.
Problem
Neural operators for PDE solution learning overfit when training data is limited, and expensive PDE solver data makes this a bottleneck; existing attention-based operators lack inherent regularization.
Approach
Rewrite the kernel integral operator with orthonormal eigenfunctions and parameterize them with neural networks; orthonormalize features via Cholesky whitening of an EMA covariance, yielding a softmax-free attention-like module with built-in orthogonal regularization.
Key results
- ONO achieves best error on NS2d (0.1195 vs 0.1213 F-FNO), Airfoil (0.0056), and Pipe (0.0034), reducing average error by 13% across benchmarks and by 31% on Pipe.
- Zero-shot super-resolution on Darcy trained at 43x43 and tested at 421x421: ONO 0.0386 vs FNO 0.3631, an 89% reduction.
- Scaling ONO to 30 layers (about 10M params) reduces Elasticity error from 0.0118 to 0.0047 and Plasticity from 0.0048 to 0.0013.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use ONO instead of the strongest listed baseline | neural operator · NS2d | l2 relative error: 0.1213 -> 0.1195 | Helped |
| use ONO instead of the strongest listed baseline | neural operator · Airfoil | l2 relative error: 0.0062 -> 0.0056 | Helped |
| use ONO instead of the strongest listed baseline | neural operator · Pipe | l2 relative error: 0.0049 -> 0.0034 | Helped |
| use ONO instead of the strongest listed baseline | neural operator · Darcy | l2 relative error: 0.0069 -> 0.0072 | Hurt |
| use ONO instead of the strongest listed baseline | neural operator · Elasticity | l2 relative error: 0.0086 -> 0.0118 | Hurt |
| use ONO instead of the strongest listed baseline | neural operator · Plasticity | l2 relative error: 0.0028 -> 0.0048 | Hurt |
| replace FNO with ONO | operator on Darcy zero-shot super-resolution at s = 421 · Darcy zero-shot super-resolution (s = 421) | l2 relative error: 0.3631 -> 0.0386 | Helped |
| replace FNO with ONO | operator on NS2d · NS2d Seen and Unseen timesteps | l2 relative error: 0.0982 -> 0.0889; 0.2446 -> 0.2143 | Helped |
| add orthogonalization instead of using no orthogonalization (ablation) | ONO on Elasticity with reduced training data · Elasticity limited-data experiment | l2 relative error: ONO: 0.0114 -> 0.0181; ONO$^{-}$: 0.0352 -> 0.0694 | Helped |
| use Nyström Transformer block instead of Linear Transformer block (ablation) | ONO · Airfoil, Elasticity, and Pipe | l2 relative error: Airfoil: 0.0079 -> 0.0056; Elasticity: 0.0137 -> 0.0118; Pipe: 0.0060 -> 0.0034 | Helped |
| use Nyström Transformer block instead of Galerkin Transformer block (ablation) | ONO · Airfoil, Elasticity, and Pipe | l2 relative error: Airfoil: 0.0122 -> 0.0056; Elasticity: 0.0133 -> 0.0118; Pipe: 0.0075 -> 0.0034 | Helped |
| use orthogonalization instead of batch or layer normalization (ablation) | ONO · Airfoil, Elasticity, and Pipe | l2 relative error: BN -> Ortho: Airfoil 0.0808 -> 0.0056, Elasticity 0.0149 -> 0.0118, Pipe 0.2151 -> 0.0034; LN -> Ortho: Airfoil 0.0288 - | Helped |
Surprising findings
- Batch normalization in place of orthogonalization gives Pipe error 0.2151, far worse than layer norm (0.0056), showing normalization choice is highly sensitive.
- ONO uses more memory than baselines (2-16 GB vs 1-9 GB), a limitation attributed to the dual-pathway architecture.
Limitations
- Higher memory requirements than baselines due to dual-pathway architecture.
- Not best on Darcy and Elasticity (GNOT best on Elasticity, LSM best on Darcy).
- Cholesky decomposition cost is cubic in the number of eigenfunctions k.
Experimental setup
Datasets: Darcy NS2d Airfoil Pipe Elasticity Plasticity
Benchmarks: Darcy NS2d Airfoil Pipe Elasticity Plasticity
Metrics: l2 relative error
Baselines: FNO Geo-FNO F-FNO U-FNO Galerkin Transformer LSM GNOT DeepONet
Builds on: Fourier Neural Operator Galerkin Transformer Neural eigenfunctions Linear Transformer Nyströmformer
Closest papers by meaning
- Neural Operators with Localized Integral and Differential Kernels (ICML 2024 · similarity 0.89)
- Amortized Fourier Neural Operators (NeurIPS 2024 · similarity 0.89)
- Data-Efficient Operator Learning via Unsupervised Pretraining and In-Context Learning (NeurIPS 2024 · similarity 0.88)
- Derivative-enhanced Deep Operator Network (NeurIPS 2024 · similarity 0.88)
- MgNO: Efficient Parameterization of Linear Operators via Multigrid (ICLR 2024 · similarity 0.88)
- Component Fourier Neural Operator for Singularly Perturbed Differential Equations (AAAI 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.