Research map / Architectures and efficiency / Attention and transformers
CViT: Continuous Vision Transformer for Operator Learning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces CViT, a neural operator combining a ViT encoder with a trainable grid-based Nadaraya-Watson coordinate embedding and query cross-attention decoder, achieving strong results on PDE benchmarks with fewer parameters.
Problem
Neural operators for PDE surrogates have architectures motivated by mathematical intuition rather than modern deep learning advances, limiting accuracy, flexibility, and resolution-independent evaluation on complex physical systems.
Approach
Combine a vision transformer encoder (patch embeddings, Perceiver temporal aggregation) with a trainable grid-based coordinate embedding that interpolates latent grid features via Nadaraya-Watson weights, queried by a cross-attention decoder, plus a theoretical Lipschitz/spectral-bias analysis of coordinate embeddings.
Key results
- On shallow-water 5-step rollout, CViT-L reaches 1.56% relative L2 error with 92M params, versus 1.89% for U-F2Net (344M).
- On incompressible Navier-Stokes, CViT-L achieves 2.35% relative L2 error, better than DPOT-H pretrained (3.79%, 1.03B params).
- On diffusion-reaction, CViT-L attains 0.68% error, below DPOT-L fine-tuned (0.73%) without rollout fine-tuning.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use CViT-L instead of U-F2Net | Shallow-water equations 5-step roll-out · Shallow-water equations | Relative L2 error: 1.89% -> 1.56% | Helped |
| Use CViT-L instead of DPOT-L (Fine-tuned) | Incompressible Navier-Stokes rollout · Incompressible Navier-Stokes (NS) | Relative L2 error: 2.78 % -> 2.35 % | Helped |
| Use CViT-L instead of DPOT-L (Fine-tuned) | Compressible Navier-Stokes rollout · Compressible Navier-Stokes (CNS) | Relative L2 error: 1.31 % -> 1.29 % | Helped |
| Use CViT-L instead of DPOT-L (Fine-tuned) | Diffusion-Reaction rollout · Diffusion-Reaction (DR) | Relative L2 error: 0.73 % -> 0.68 % | Helped |
| Use smaller input patch sizes (ablation) | CViT-L · Shallow-water equations | test error | Helped |
| Replace the grid-based decoder coordinate embedding with a small MLP or random Fourier features (ablation) | CViT-L · Shallow-water equations | test error | Hurt |
| Match latent grid resolution to the highest evaluation resolution (ablation) | CViT-L · Shallow-water equations | test error | Helped |
| Use too small a beta for interpolating grid features (ablation) | CViT-L · Shallow-water equations | test error | Hurt |
| Increase the number of decoder cross-attention layers (ablation) | CViT · PDE benchmarks | test error | No clear effect |
| Increase grid-feature latent dimension or decoder attention heads (ablation) | CViT · Shallow-water equations | test error | Helped |
Surprising findings
- CViT was trained with a one-step loss and no rollout fine-tuning, yet it beats rollout-pretrained DPOT on rollout evaluation.
- CViT-S (13M params) achieves accuracy comparable to the 1.03B-parameter DPOT-H pretrained model on Navier-Stokes.
Limitations
- Self-attention scales quadratically with token count, limiting high-resolution or 3D scaling.
- Transformer models generally need large training datasets and may struggle in data-scarce settings.
- Currently handles only regular grid inputs treated as image sequences, not unstructured meshes or point clouds.
Experimental setup
Datasets: PDEArena shallow-water PDEArena Navier-Stokes PDEBench compressible Navier-Stokes PDEBench diffusion-reaction Hao et al. advection benchmark
Benchmarks: advection of discontinuous waveforms shallow-water equations Navier-Stokes compressible Navier-Stokes diffusion-reaction
Metrics: relative L2 error total variation error
Baselines: DeepONet NoMaD FNO FFNO UNO U-Net_att U-F2Net Dilated ResNet GNOT OFormer DPOT MPP
Builds on: Fourier Neural Operator DeepONet OFormer Perceiver Vision Transformer
Closest papers by meaning
- Geometry Aware Operator Transformer as an efficient and accurate neural surrogate for PDEs on arbitrary domains (NeurIPS 2025 · similarity 0.89)
- Adaptive Coordinate Transforms for Neural Operators (NeurIPS 2026 · similarity 0.87)
- Data-Efficient Operator Learning via Unsupervised Pretraining and In-Context Learning (NeurIPS 2024 · similarity 0.87)
- RIGNO: A Graph-based Framework For Robust And Accurate Operator Learning For PDEs On Arbitrary Domains (NeurIPS 2025 · similarity 0.87)
- Smooth Piecewise Cutting for Neural Operator to Handle Discontinuities and Sharp Transitions (NeurIPS 2026 · similarity 0.86)
- CALM-PDE: Continuous and Adaptive Convolutions for Latent Space Modeling of Time-dependent PDEs (NeurIPS 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.