Research map / Architectures and efficiency / Attention and transformers
Positional Knowledge is All You Need: Position-induced Transformer (PiT) for Operator Learning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Position-induced Transformer (PiT) with position-attention, a Softmax over negative squared pairwise distances of mesh points that ignores input function values, for PDE operator learning.
Problem
Transformer-based neural operators for PDE operator learning rely on content-based self-attention, which is computationally expensive (memory and time scale with each batch's input values) and has limited interpretability.
Approach
Replace content-based self-attention with position-attention whose weights come only from a Softmax of negative squared distances between sampling points, making the kernel input-independent and convergent under mesh refinement; adds cross and local variants for down/upsampling and a local receptive field.
Key results
- PiT achieves the best reported relative errors on Darcy2D (0.00485), Vorticity (0.1140), Elasticity (0.00649), NACA (0.00480) and ShockTube (0.0122) among compared methods.
- On Darcy2D discretization convergence tests, PiT43's error rises from 0.97% to 4.50% at 421^2 resolution, 48% lower than FNO++ (8.67%).
- On Darcy2D, PiT with d_v=32 uses about 20,000 parameters and reaches 0.00808 error, while FNO, OFormer and Galerkin Transformer have over 2 million parameters.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use PiT with position-attention instead of FNO++ | operator learning model · InviscidBurgers | relative l1 error: 0.00995 -> 0.0136 | Hurt |
| use PiT with position-attention instead of FNO | operator learning model · ShockTube | relative l1 error: 0.0156 -> 0.0122 | Helped |
| use PiT with position-attention instead of FNO++ | operator learning model · Darcy2D | relative l2 error: 0.00509 -> 0.00485 | Helped |
| use PiT with position-attention instead of ONO | operator learning model · Vorticity | relative l2 error: 0.1195 -> 0.1140 | Helped |
| use PiT with position-attention instead of GNOT | operator learning model · Elasticity | relative l2 error: 0.00865 -> 0.00649 | Helped |
| use PiT with position-attention instead of ONO | operator learning model · NACA | relative l2 error: 0.0056 -> 0.00480 | Helped |
| use PiT instead of FNO++ when trained on 43^2 | Darcy2D operator · Darcy2D, trained on 43^2 and tested on 421^2 | prediction error: 8.67% -> 4.50% | Helped |
| replace all position-attention layers with vanilla self-attention (ablation) | PiT · InviscidBurgers | relative l1 error: 0.0136 -> 0.016 | Hurt |
| replace all position-attention layers with vanilla self-attention (ablation) | PiT · ShockTube | relative l1 error: 0.0122 -> 0.0259 | Hurt |
| replace Processor position-attention layers with vanilla self-attention (ablation) | PiT · Elasticity | relative l2 error: 0.00649 -> 0.169 | Hurt |
| combine self-attention with position-attention (ablation) | PiT · InviscidBurgers and ShockTube | relative l1 error: InviscidBurgers: 0.0136 -> 0.00816; ShockTube: 0.0122 -> 0.0179 | Mixed |
| replace lambda with lambda^2 to ensure non-negativity (ablation) | PiT · InviscidBurgers, ShockTube, Darcy2D, Vorticity, Elasticity, and NACA | relative error: InviscidBurgers: 0.0136 -> 0.0273; ShockTube: 0.0154 -> 0.0122; Darcy2D: 0.00485 -> 0.0102; Vorticity: 0.5757949 -> 0.11 | Mixed |
Surprising findings
- Position-attention without input-value dependence outperforms self-attention variants on the benchmarks, suggesting self-attention may be unnecessary for Transformer-based operator learning.
- Combining self-attention with position-attention (Self-PiT) did not consistently improve accuracy over PiT while adding computational cost.
Limitations
- Global position-attention still has quadratic cost in latent mesh size, so a sufficiently large latent mesh is needed for accuracy.
- Requires a non-negativity constraint on lambda (tan parameterization or constrained optimization); the lambda^2 parameterization performed worse on several benchmarks.
- Self-attention variant OOM issues on larger benchmarks limit the direct comparison.
Experimental setup
Datasets: InviscidBurgers ShockTube Darcy2D Vorticity (Navier-Stokes) Elasticity NACA airfoil transonic flow
Benchmarks: InviscidBurgers ShockTube Darcy2D Vorticity Elasticity NACA
Metrics: relative l1 error relative l2 error parameter count training time per epoch
Baselines: DeepONet shift-DeepONet FNO FNO++ Geo-FNO Galerkin Transformer OFormer GNOT ONO
Builds on: Transformer Fourier neural operator DeepONet Galerkin Transformer OFormer
Closest papers by meaning
- Adaptive Coordinate Transforms for Neural Operators (NeurIPS 2026 · similarity 0.88)
- Geometry Aware Operator Transformer as an efficient and accurate neural surrogate for PDEs on arbitrary domains (NeurIPS 2025 · similarity 0.87)
- Pretraining Codomain Attention Neural Operators for Solving Multiphysics PDEs (NeurIPS 2024 · similarity 0.87)
- Improved Operator Learning by Orthogonal Attention (ICML 2024 · similarity 0.86)
- CViT: Continuous Vision Transformer for Operator Learning (ICLR 2025 · similarity 0.86)
- Data-Efficient Operator Learning via Unsupervised Pretraining and In-Context Learning (NeurIPS 2024 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.