Research map / Probabilistic and scientific modeling / Dynamical systems
Universal Approximation of Mean-Field Models via Transformers
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Shows empirically that transformers learn permutation-equivariant mean-field vector fields (Cucker-Smale, fish milling, two-layer net training) and proves that an expected (measure-lifted) transformer approximates the mean-field vector field and its continuity-equation solution in W1.
Problem
Learning and simulating mean-field dynamics of interacting particle systems with permutation-equivariant dynamics, where the vector field is defined on an infinite-dimensional space of probability measures, so standard finite-sequence transformers do not directly apply.
Approach
Lifts a finite-sequence transformer to a map on measures by taking its expectation over i.i.d. samples from the measure (the "expected transformer"), then bounds the approximation error of the mean-field vector field and the resulting continuity-equation solution in W1 using Wasserstein empirical-measure convergence and Gronwall's lemma.
Key results
- On Cucker-Smale vector-field learning, the transformer achieves MSE of (1.9 ± 0.3)×10^-6, versus 6.7×10^-6 for FNN and 3.0×10^-5 for kernel regression.
- On fish milling, the transformer achieves MSE of (2.2 ± 0.0)×10^-2, versus 1.2×10^-1 for the best GraphConv baseline (m=3).
- Trajectories of 30 particles simulated with the transformer (trained on 20) track the true Cucker-Smale dynamics with 2-norm errors generally below 10^-4.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use the Transformer instead of TransformerConv m=20 | Cucker-Smale vector-field approximation · Cucker-Smale | mean-squared error: (3.3 \pm 0.1) \times 10^{-6} -> (1.9 \pm 0.3) \times 10^{-6} | Helped |
| Use the Transformer instead of TransformerConv m=3 | Fish Milling vector-field approximation · Fish Milling | mean-squared error: (6.5 \pm 1.7) \times 10^{-2} -> (2.2 \pm 0.0) \times 10^{-2} | Helped |
Surprising findings
- The transformer trained on 20 particles generalizes to simulations with 30 particles.
- Transformer-based updates for two-layer network training keep parameter differences from gradient descent at most 5×10^-5 after 500 iterations, even though the transformer computes no gradients.
Limitations
- The expected transformer may not be globally Lipschitz, so uniqueness of solutions for the approximate continuity equation is only guaranteed under local conditions.
- Theoretical bounds depend on Lipschitz and regularity assumptions on the vector field and are only approximated numerically via Gibbs sampling for the expected transformer.
- Fish milling data is incomplete with varying particle counts and the MSE gap comparisons use a single experimental dataset.
Experimental setup
Datasets: Cucker-Smale synthetic simulations fish milling experimental dataset two-layer neural network training dynamics (synthetic)
Benchmarks: Cucker-Smale synthetic Fish milling
Metrics: mean-squared error trajectory error (2-norm) Wasserstein-1 distance (theoretical)
Baselines: Cylindrical NN GraphConv TransformerConv FNN Kernel regression
Builds on: transformer Gronwall's lemma
Closest papers by meaning
- Emergence of meta-stable clustering in mean-field transformer models (ICLR 2025 · similarity 0.87)
- Improved Particle Approximation Error for Mean Field Neural Networks (NeurIPS 2024 · similarity 0.87)
- Quantifying Concentration Phenomena of Mean-Field Transformers in the Low-Temperature Regime (NeurIPS 2026 · similarity 0.86)
- A multiscale analysis of mean-field transformers in the moderate interaction regime (NeurIPS 2025 · similarity 0.86)
- Training-Induced Escape from Token Clustering in a Mean-Field Formulation of Transformers (NeurIPS 2026 · similarity 0.86)
- Parametric model reduction of mean-field and stochastic systems via higher-order action matching (NeurIPS 2024 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.