Research map / Architectures and efficiency / Attention and transformers
Functional Attention: From Pairwise Affinities to Functional Correspondences
Jiefang Xiao, Maolin Gao, Simon Weber, Guandao Yang, Daniel Cremers
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| replace softmax attention with Functional Attention (least-squares operator on learned basis) | Transolver-style PDE solver · Six PDE benchmarks | relative L2: Elasticity 0.64 -> 0.50; Darcy 0.57 -> 0.42 | Improved |
| use FuncAttn instead of Transolver on Pipe | FuncAttn · Pipe | relative L2: 0.31 -> 0.29 (LNO 0.25) | No clear effect |
| use learned free basis instead of fixed Fourier basis (ablation) | FuncAttn · Airfoil | relative L2: 0.51 -> 0.43 | Improved |
| add orthogonality constraint to learnable basis (ablation) | FuncAttn · Airfoil | relative L2: 0.43 -> 0.50 | Hurt |
| learnable orthogonal basis instead of free learnable basis in scaled dot-product attention (ablation) | Scaled dot-product attention · Airfoil | relative L2: 0.53 -> 0.65 | Hurt |
| increase number of bases from 16 to 64 (ablation) | FuncAttn · Elasticity | relative L2: 0.65 -> 0.50 | Improved |
| increase number of bases from 64 to 512 (ablation) | FuncAttn · Elasticity | relative L2: 0.50 -> 0.56 | Hurt |
| increase number of bases beyond 64 on Darcy (ablation) | FuncAttn · Darcy | relative L2: 0.42 -> 0.41 (k=512) | No clear effect |
| use transpose projection instead of Moore-Penrose pseudo-inverse (ablation) | FuncAttn · stability of training | gradient stability | Improved |
| Tikhonov-stabilized pseudo-inverse instead of transpose (ablation) | FuncAttn · condition number | condition number | Hurt |
| share learnable basis modules across layers | FuncAttn · PDE benchmarks | accuracy | Improved |
| FuncAttn vs Transolver on OOD airfoil design | FuncAttn · AirfRANS OOD angles | CL relative error: 22.8 -> 13.3 | Improved |
Closest papers by meaning
- FAME: Adaptive Functional Attention with Expert Routing for Function-on-Function Regression (NeurIPS 2025 · similarity 0.88)
- Nonlocal Attention Operator: Materializing Hidden Knowledge Towards Interpretable Physics Discovery (NeurIPS 2024 · similarity 0.86)
- Neural Interpretable PDEs: Harmonizing Fourier Insights with Attention for Scalable and Interpretable Physics Discovery (ICML 2025 · similarity 0.86)
- Function graph transformers universally approximate operators between function spaces (NeurIPS 2026 · similarity 0.85)
- Function Encoders: A Principled Approach to Transfer Learning in Hilbert Spaces (ICML 2025 · similarity 0.85)
- Sparse Attention as Compact Kernel Regression (NeurIPS 2026 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
