Research map / Architectures and efficiency / Attention and transformers
Can Transformers Do Enumerative Geometry?
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes DynamicFormer, a multi-modal Transformer with a new Dynamic Range Activator (DRA) to regress psi-class intersection numbers across genera, with conformal uncertainty and interpretability probes for Virasoro constraints and large-genus asymptotics.
Problem
Computing psi-class intersection numbers on the moduli space of curves via brute-force recursion has factorial complexity; standard neural nets struggle to model their recursive, high-variance, factorially growing values out of distribution.
Approach
A new activation function, DRA(x) = x + a sin^2(x/b) + c cos(bx) + d tanh(bx), with learnable parameters, is used inside a multi-modal Transformer (DynamicFormer) that takes the B tensor (COO sequence) and partition set, combined with conformal prediction using a dynamic sliding window and probing/causal tracing to analyze learned representations.
Key results
- DynamicFormer with DRA reaches R^2 of 95.8 and conformal width 3.42 in the OOD regime, versus 71.5 for ReLU MLP, 74.7 for GLU and 82.9 for Snake.
- In the ID setting, R^2 is above 98.7 for all tested (g,n) pairs with about 90% empirical coverage at alpha=0.1.
- A non-linear probe recovers the exponential growth constant encoding with higher R^2 than a linear probe, and the subleading coefficients reach R^2 = 0.98 (non-linear) versus 0.63 (linear).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use DRA instead of ReLU, GLU, or Snake activation | DynamicFormer · ψ-class intersection numbers, OOD (g=14–17) | R²: 71.5 ; 74.7 ; 82.9 -> 95.8 | Helped |
| use DRA instead of ReLU, GLU, or Snake activation | DynamicFormer · ψ-class intersection numbers, OOD (g=14–17) | CW: 9.73 ; 8.34 ; 6.55 -> 3.42 | Helped |
| use DynamicFormer instead of a DRA or Snake MLP | ψ-class intersection number prediction · OOD (g=14–17) | R²: 76.5 ; 69.3 -> 95.8 | Helped |
| use DynamicFormer instead of a DRA or Snake MLP | ψ-class intersection number prediction · OOD (g=14–17) | CW: 7.73 ; 8.44 -> 3.42 | Helped |
| add [DYN] register tokens and Talking Modalities loss (ablation) | DynamicFormer · ψ-class intersection numbers, OOD (g=14–17) | R²: 91.1 -> 95.8 | Helped |
| add [DYN] register tokens and Talking Modalities loss (ablation) | DynamicFormer · ψ-class intersection numbers, OOD (g=14–17) | CW: 4.91 -> 3.42 | Helped |
| add [DYN] register tokens (ablation) | DynamicFormer · ψ-class intersection numbers, OOD (g=14–17) | R²: 91.1 -> 94.6 | Helped |
| add [DYN] register tokens (ablation) | DynamicFormer · ψ-class intersection numbers, OOD (g=14–17) | CW: 4.91 -> 4.05 | Helped |
| add Talking Modalities loss to [DYN] tokens (ablation) | DynamicFormer · ψ-class intersection numbers, OOD (g=14–17) | R²: 94.6 -> 95.8 | Helped |
| add Talking Modalities loss to [DYN] tokens (ablation) | DynamicFormer · ψ-class intersection numbers, OOD (g=14–17) | CW: 4.05 -> 3.42 | Helped |
Surprising findings
- Causal tracing shows that the B initial datum has only weak causal effect on predicted intersection numbers, yet it appears causally linked to encoding the asymptotic constant A=2/3.
- The C tensor was excluded from inputs because B was found to have a larger effect on the intersection numbers.
Limitations
- Gains from individual components (DRA, DYN+TM) are reported on a single OOD metric without variance, and some results are from a small set of examples.
- Interpretability conclusions (Virasoro constraints, asymptotic encoding) rely on correlational and probe evidence rather than formal proof.
- Training is limited to genus up to 13, with the claim that the model learned Virasoro constraints not strictly verified beyond Dilaton equation.
Experimental setup
Datasets: psi-class intersection numbers computed by brute-force recursion, genus up to 13
Benchmarks: psi-class intersection numbers ID (genus 1-13) psi-class intersection numbers OOD (genus 14-17)
Metrics: R^2 empirical coverage conformal width
Baselines: ReLU Tanh GLU Snake Swish PolyNorm KAN MLP
Builds on: Snake activation Kolmogorov-Arnold Networks Dynamic Positional Bias Barlow Twins Principal Neighbourhood Aggregation
Closest papers by meaning
- (How) Can Transformers Predict Pseudo-Random Numbers? (ICML 2025 · similarity 0.83)
- Predictive Geometry of Hidden Trajectories in Transformers (NeurIPS 2026 · similarity 0.83)
- Hide & Seek: Transformer Symmetries Obscure Sharpness & Riemannian Geometry Finds It (ICML 2025 · similarity 0.82)
- Exact Expressive Power of Transformers with Padding (NeurIPS 2025 · similarity 0.81)
- Machine Learning meets Algebraic Combinatorics: A Suite of Datasets Capturing Research-level Conjecturing Ability in Pure Mathematics (ICML 2025 · similarity 0.81)
- Arithmetic Transformers Can Length-Generalize in Both Operand Length and Count (ICLR 2025 · similarity 0.81)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.