Research map / Probabilistic and scientific modeling / Differential equations and PDEs
PINNsFormer: A Transformer-Based Framework For Physics-Informed Neural Networks
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes PINNsFormer, a Transformer-based PINN that turns point-wise inputs into pseudo time sequences and uses a sequential PINN loss plus a Wavelet activation to solve PDEs with better generalization than MLP-based PINNs.
Problem
Conventional MLP-based PINNs ignore temporal dependencies, so they fail to propagate initial-condition constraints and suffer failure modes (overly smooth or naive solutions) on high-frequency/multiscale PDEs and struggle on high-dimensional PDEs.
Approach
Transform each spatiotemporal point into a pseudo sequence of future time steps, process it with a Transformer encoder-decoder using a Wavelet activation (learnable sin/cos combination), and apply the PINN residual, boundary and initial losses across the sequence.
Key results
- On convection (beta=50), PINNsFormer reaches rMAE 0.023 and rRMSE 0.027 versus 0.674-0.778 rMAE for PINNs, QRes and FLS.
- On 1D-wave, PINNsFormer+NTK achieves rMAE 0.054 and rRMSE 0.058, improving over PINNs+NTK (0.140 rMAE).
- On 2D Navier-Stokes, PINNsFormer gets rMAE 0.384 and rRMSE 0.280 versus 3.98 rMAE for the best baseline (FLS).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| replace pointwise MLP PINNs with PINNsFormer’s pseudo-sequence Transformer framework | PINNs · Convection | rMAE: 0.778 -> 0.023 | Helped |
| replace pointwise MLP PINNs with PINNsFormer’s pseudo-sequence Transformer framework | PINNs · 1D-Reaction | rMAE: 0.982 -> 0.015 | Helped |
| add NTK to PINNsFormer | PINNsFormer · 1D-Wave | rMAE: 0.270 -> 0.054 | Helped |
| replace pointwise MLP PINNs with PINNsFormer’s pseudo-sequence Transformer framework | PINNs · Navier-Stokes | rMAE: 13.08 -> 0.384 | Helped |
| use ReLU, Sigmoid, or Sin instead of Wavelet activation (ablation) | PINNsFormer · Convection | rMAE: 0.023 vs 1.001; 1.112; 1.074 | Hurt |
| use ReLU, Sigmoid, or Sin instead of Wavelet activation (ablation) | PINNsFormer · 1D-Reaction | rMAE: 0.015 vs 0.994; 0.991; 0.017 | Mixed |
| add LayerNorm to the encoder/decoder activation setup (ablation) | PINNsFormer with ReLU · Convection | rMAE: 1.001 -> 1.001 | Mixed |
| add LayerNorm to the encoder/decoder activation setup (ablation) | PINNsFormer with Wavelet · Convection | rMAE: 0.023 -> NaN | Hurt |
| increase sequence length from k=3 to k=5 (ablation) | PINNsFormer with Δt=1e-4 · 1D-Reaction | relative-ℓ2 error: 0.997 -> 0.030 | Helped |
| increase sequence length after failure mode is mitigated (ablation) | PINNsFormer · 1D-Reaction | relative-ℓ2 error: 0.030; 0.029; 0.046 | Mixed |
| use Δt=1e-1 rather than intermediate Δt values (ablation) | PINNsFormer · 1D-Reaction | relative-ℓ2 error: 0.044; 0.514; 0.743; 0.731 | Hurt |
| use Δt=1e-5 rather than intermediate Δt values (ablation) | PINNsFormer · 1D-Reaction | relative-ℓ2 error: 0.977; 0.026; 0.977; 0.021 | Mixed |
Surprising findings
- Wavelet+LayerNorm diverged (NaN) on convection, indicating LayerNorm can cause convergence issues with Wavelet activation.
- Hyperparameter sensitivity study shows failure modes at extreme delta t values (1e-1 and 1e-5) and some k values, so PINNsFormer is not uniformly robust to pseudo-sequence settings.
Limitations
- Computational and memory overhead grows roughly linearly with pseudo-sequence length k relative to PINNs.
- Sensitive to choice of delta t and k; some settings fail to mitigate failure modes.
- Universal approximation claim for Wavelet is only a proof sketch and relies on infinite width.
Experimental setup
Datasets: convection PDE 1D-reaction PDE 1D-wave PDE 2D Navier-Stokes PDE
Benchmarks: convection PDE 1D-reaction PDE 1D-wave PDE 2D Navier-Stokes PDE
Metrics: rMAE rRMSE training loss
Baselines: PINNs QRes First-Layer Sine (FLS) NTK
Builds on: PINNs Transformer Neural Tangent Kernel
Closest papers by meaning
- Sub-Sequential Physics-Informed Learning with State Space Model (ICML 2025 · similarity 0.91)
- PINNsAgent: Automated PDE Surrogation with Large Language Models (ICML 2025 · similarity 0.90)
- Challenges in Training PINNs: A Loss Landscape Perspective (ICML 2024 · similarity 0.89)
- Efficient Error Certification for Physics-Informed Neural Networks (ICML 2024 · similarity 0.89)
- Per-Loss Adapters for Gradient Conflict in Physics-Informed Neural Networks (NeurIPS 2026 · similarity 0.89)
- Parameterized Physics-informed Neural Networks for Parameterized PDEs (ICML 2024 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.