Research map / Architectures and efficiency / Attention and transformers
Nonlocal Attention Operator: Materializing Hidden Knowledge Towards Interpretable Physics Discovery
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Nonlocal Attention Operator (NAO), an attention-based neural operator whose attention acts as a data-dependent kernel map that jointly solves forward PDE surrogate and inverse kernel discovery across multiple physical systems.
Problem
Learning physical operators from few function pairs is a severely ill-posed inverse PDE problem; existing neural operators act as black-box forward surrogates and do not reveal the hidden kernel or generalize well to unseen systems.
Approach
Shows attention is equivalent to a double integral operator with a data-dependent kernel; builds a kernel map from (u,f) token pairs across multiple training systems, which yields a space of identifiability (data-adaptive RKHS) that regularizes the inverse problem and enables zero-shot generalization to new kernels and resolutions.
Key results
- On radial kernel OOD1 test with d=302, d_k=10, NAO achieves 8.10% operator error and 10.02% kernel error vs. 25.81% and 28.80% for Discrete-NAO (kernel OOD error for Discrete-NAO is 28.80%).
- On Mechanical MNIST with d=100, NAO reaches 4.75% ID and 5.58% OOD error vs. 6.34% and 6.01% for Discrete-NAO, with ~303k vs 7.35M parameters.
- On Darcy g->p, NAO gives 3.19% vs 3.28% for Discrete-NAO at 9000 samples, using 189k vs 662k parameters.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use continuous linear attention instead of the discrete mixer (ablation) | NAO · Radial kernel learning OOD1 | operator test error: 25.81% -> 8.10% | Helped |
| use continuous linear attention instead of the discrete mixer (ablation) | NAO · Radial kernel learning ID | operator test error: 1.33% -> 1.48% | Hurt |
| use continuous linear attention instead of the discrete mixer (ablation) | NAO · Radial kernel learning OOD1 | kernel test error: 28.80% -> 10.02% | Helped |
| replace linear attention with softmax attention (ablation) | NAO · Radial kernel learning ID and OOD1 | operator test error: 1.48% -> 13.45%; 8.10% -> 66.06% | Hurt |
| use input u only rather than both u and f (ablation) | NAO · Radial kernel learning ID and OOD1 | kernel test error: 5.40% -> 20.46%; 10.02% -> 74.03% | Hurt |
| replace the attention kernel map with an autoencoder (ablation) | NAO · Radial kernel learning OOD1 | operator test error: 8.10% -> 1041.49% | Hurt |
| increase query-key feature size from d_k=10 to d_k=20 (ablation) | NAO · Radial kernel learning ID | kernel test error: 5.40% -> 4.63% | Helped |
| increase query-key feature size from d_k=10 to d_k=20 (ablation) | NAO · Radial kernel learning OOD1 | kernel test error: 10.02% -> 9.14% | Helped |
| train on diverse sine, cosine, and polynomial tasks rather than sine-only tasks (ablation) | NAO · Radial kernel OOD1 | kernel test error: 9.14% -> 6.92% | Helped |
| train on diverse sine, cosine, and polynomial tasks rather than sine-only tasks (ablation) | NAO · Radial kernel ID | kernel test error: 4.03% -> 5.04% | Hurt |
| train on diverse sine, cosine, and polynomial tasks rather than sine-only tasks (ablation) | NAO · Radial kernel OOD2 | kernel test error: 329.66% -> 10.48% | Helped |
| train on one sine task instead of multiple tasks (ablation) | NAO · Radial kernel ID | kernel test error: 4.03% -> 100.31% | Hurt |
Surprising findings
- Discrete attention (Discrete-NAO) yields small operator error on ID tests but large kernel error and spurious oscillations, showing good forward fit does not imply correct inverse kernel recovery.
- Softmax-NAO and NAO-u perform much worse than the linear continuous NAO, with NAO-u kernel errors of 74% on OOD test.
Limitations
- Experiments limited to learning from fewer than 500 similar physical systems; no cross-type physics coverage.
- Kernel recovery near zero-measure regions (|x-y|=0) and near domain boundaries is unidentifiable from data.
- OOD generalization degrades for kernels far from training distribution (e.g., Gaussian OOD2 in sine-only training gives >200% error).
Experimental setup
Datasets: Synthetic radial nonlocal diffusion kernels Darcy flow synthetic data (500 microstructures) Mechanical MNIST (heterogeneous Neo-Hookean material)
Benchmarks: Radial nonlocal kernel learning (ID/OOD test) Darcy flow solution operator learning Mechanical MNIST
Metrics: relative L2 operator (forward) error relative L2 kernel (inverse) error
Baselines: Discrete-NAO Softmax-NAO NAO-u AFNO Autoencoder
Builds on: AFNO Fourier neural operator linear attention neural operator OFormer
Closest papers by meaning
- Neural Interpretable PDEs: Harmonizing Fourier Insights with Attention for Scalable and Interpretable Physics Discovery (ICML 2025 · similarity 0.95)
- Improved Operator Learning by Orthogonal Attention (ICML 2024 · similarity 0.86)
- Data-Efficient Operator Learning via Unsupervised Pretraining and In-Context Learning (NeurIPS 2024 · similarity 0.84)
- Pretraining Codomain Attention Neural Operators for Solving Multiphysics PDEs (NeurIPS 2024 · similarity 0.84)
- Universal Physics Transformers: A Framework For Efficiently Scaling Neural Operators (NeurIPS 2024 · similarity 0.84)
- Dissecting the Interplay of Attention Paths in a Statistical Mechanics Theory of Transformers (NeurIPS 2024 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.