Research map / Probabilistic and scientific modeling / Differential equations and PDEs
Unisolver: PDE-Conditional Transformers Towards Universal Neural PDE Solvers
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Unisolver, a PDE-conditional Transformer that embeds complete PDE components (LLM-embedded equation symbols, coefficients, forces, geometry, boundaries) as domain-wise and point-wise conditions in decoupled hidden subspaces, achieving state-of-the-art on three large-scale neural PDE benchmarks.
Problem
Neural PDE solvers are typically restricted to a few PDE instances (fixed equation and limited coefficients), so they fail to generalize across diverse PDEs and to unseen coefficients, forces, or equation forms.
Approach
Classify complete PDE components into domain-wise (equation symbols via LLaMA-3 embedding, coefficients, boundary type) and point-wise (force, geometry mask, boundary value fields) conditions; embed them as deep conditions that scale/shift/select Transformer features, with hidden channels split into decoupled subspaces for each category.
Key results
- On HeterNS, Unisolver (4.1M params) reduces in-distribution viscosity error to 0.0098 vs. 0.0156 for the best baseline (ViT), a 37.2% promotion; zero-shot force promotion averages 66.7%.
- On 1D time-dependent PDEs, Unisolver (19M) averages 27.1% lower zero-shot error than PDEformer (22M) across four zero-shot settings.
- On 2D mixed PDEs, Unisolver improves average relative L2 from 5.50 to 4.54 (17.5%) over DPOT, winning 11 of 12 in-distribution test sets.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Condition on diverse PDE components with Unisolver | PDE solvers · HeterNS viscosity and force in-distribution and zero-shot tests | relative L2: 37.2%; 27.2%; 27.7%; 66.7% | Helped |
| Use complete PDE-component conditioning | 1D time-dependent PDE solver · 1D time-dependent PDEs, including zero-shot Burgers and Advection | relative L2: 52.0%; 31.0%; 30.9%; 23.9%; 22.5% | Helped |
| Jointly train on 12 mixed PDE datasets | 2D mixed PDE solver · 2D mixed PDEs average | relative L2: 5.50→ 4.54 | Helped |
| Add LLM embeddings for equation symbols (ablation) | Unisolver · 1D time-dependent PDEs, in-distribution | relative L2: 0.0295 -> 0.0277; 5.76% | Helped |
| Replace LLM embeddings with orthogonal random vectors (ablation) | Unisolver · 1D time-dependent PDEs, in-distribution | relative L2: 0.0277 -> 0.0290 | Hurt |
| Add subspace decoupling for condition modeling (ablation) | Unisolver · 1D time-dependent PDEs zero-shot generalization | relative L2: 5.45% average drop when removed | Helped |
| Use decoupled condition modeling instead of direct concatenation (ablation) | Unisolver · 1D time-dependent PDEs | relative L2: 21.0% average drop when concatenating | Helped |
| Use LLM embeddings instead of encoding only the count of non-zero terms (ablation) | Unisolver · 1D PDEs, in-distribution | relative L2: 0.0285 -> 0.0277 | Helped |
| Encode the accurate PDE expression rather than an altered wrong expression (ablation) | Unisolver · 1D PDEs, in-distribution | relative L2: 0.0289 -> 0.0277 | Helped |
| Use LLM embeddings instead of a manually constructed representation (ablation) | Unisolver · 1D time-dependent PDEs | relative L2: 4.23% average improvement | Helped |
| Jointly train across polynomial orders instead of training independently (ablation) | Unisolver · 1D PDEs polynomial orders 1, 2, and 3 | relative L2: 29.9%; 36.5%; 43.7% | Helped |
| Pretrain on polynomial orders up to 2 and fine-tune for 200 epochs on order 3 | Unisolver · Unseen polynomial order 3, 5000 fine-tuning examples | relative L2: 0.3308 -> 0.1624; 50.9% | Helped |
Surprising findings
- Embedding wrong or randomized equation information with the LLM degrades performance, and even random orthogonal vectors are worse than or comparable to no LLM embedding in some settings.
- Removing LLM embeddings is worse than replacing them with random vectors on 1D PDEs, though gains are only around 5%.
Limitations
- Evaluation limited to specific benchmarks and PDE families (mostly 1D and 2D time-dependent), with generalization to truly unseen PDE types only partially tested.
- Relies on LaTeX equation formulation and complete PDE component specification, which may be unavailable in real-world settings (partially addressed via learnable tokens).
Experimental setup
Datasets: HeterNS (self-generated) 1D time-dependent PDEs (PDEformer) 2D mixed PDEs (DPOT collection: FNO-NS, PDEBench, PDEArena, CFDBench) PDEBench Burgers and Advection (zero-shot)
Benchmarks: HeterNS 1D time-dependent PDEs 2D mixed PDEs PDEBench Burgers PDEBench Advection
Metrics: relative L2
Baselines: FNO PINO ViT FactFormer ICON MPP PDEformer DPOT
Builds on: DiT PDEformer DPOT FNO Transolver
Closest papers by meaning
- Vectorized Conditional Neural Fields: A Framework for Solving Time-dependent Parametric Partial Differential Equations (ICML 2024 · similarity 0.89)
- Breakeven complexity: A new perspective on neural partial differential equation solvers (NeurIPS 2026 · similarity 0.89)
- Text2PDE: Latent Diffusion Models for Accessible Physics Simulation (ICLR 2025 · similarity 0.87)
- HAMLET: Graph Transformer Neural Operator for Partial Differential Equations (ICML 2024 · similarity 0.87)
- Transolver++: An Accurate Neural Solver for PDEs on Million-Scale Geometries (ICML 2025 · similarity 0.87)
- MC^2: Monte Carlo Correction for Fast Elliptic PDE Solving (NeurIPS 2026 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.