Research map / Architectures and efficiency / RNNs and normalization
EquiformerV2: Improved Equivariant Transformer for Scaling to Higher-Degree Representations
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Scales the Equiformer equivariant transformer to higher-degree irreps by swapping SO(3) convolutions for eSCN SO(2) convolutions, plus attention re-normalization, separable S^2 activation and separable layer normalization.
Problem
Equivariant Transformers for 3D atomistic systems are limited to small maximum degree L_max because SO(3) tensor products are computationally expensive; it is unclear whether such architectures scale well to higher degrees.
Approach
Replace SO(3) convolutions in Equiformer with eSCN SO(2) convolutions to enable high L_max, then add three architectural changes (attention re-normalization via an extra LayerNorm, separable S^2 activation that keeps degree-0 channels separate, and separable layer normalization that preserves relative magnitudes across degrees) so that higher degrees actually improve accuracy.
Key results
- On OC20 S2EF-All+MD, EquiformerV2 (λE=4, 153M) improves S2EF force MAE by 9% and energy MAE by 4% over prior state of the art (eSCN/SCN baselines).
- On OC22, EquiformerV2 trained only on OC22 (λE=4, λF=100) outperforms GemNet-OC trained on OC20+OC22 on all tasks, e.g., S2EF test force MAE 21.58 vs 26.9 meV/Å (ID).
- In AdsorbML, EquiformerV2 (λE=4, 153M) reaches 85.41% success at k=1 versus 77.80% for SCN-MD-Large, and at k=2 is more accurate than all baselines at k=5.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add attention re-normalization (ablation) | EquiformerV2 · OC20 S2EF-2M validation | Energy MAE: 286 -> 279 | Helped |
| add attention re-normalization (ablation) | EquiformerV2 · OC20 S2EF-2M validation | Force MAE: 21.85 -> 21.86 | No clear effect |
| replace gate activation with unseparated S² activation (ablation) | EquiformerV2 · OC20 S2EF-2M validation | convergence | Hurt |
| replace gate activation with separable S² activation (ablation) | EquiformerV2 · OC20 S2EF-2M validation | Force MAE: 21.86 -> 20.77 | Helped |
| replace equivariant layer normalization with separable layer normalization (ablation) | EquiformerV2 · OC20 S2EF-2M validation | Force MAE: 20.77 -> 20.46 | Helped |
| increase training from 12 to 20 epochs (ablation) | EquiformerV2 · OC20 S2EF-2M validation | Force MAE: 20.46 -> 19.72 | Helped |
| increase Lmax from 4 to 6 (ablation) | EquiformerV2 · OC20 S2EF-2M validation | Force MAE: 21.37 -> 20.46 | Helped |
| increase Lmax from 6 to 8 (ablation) | EquiformerV2 · OC20 S2EF-2M validation | Energy MAE: 285 -> 279 | Helped |
| increase Mmax from 2 to 6 (ablation) | EquiformerV2 · OC20 S2EF-2M validation | Energy MAE: 285 -> 278 | Helped |
| increase Transformer blocks from 8 to 12 (ablation) | EquiformerV2 · OC20 S2EF-2M validation | Force MAE: 21.18 -> 20.46 | Helped |
| replace Equiformer convolutions with eSCN convolutions and use Lmax=6 | EquiformerV2 · OC20 S2EF-2M validation | Force MAE: 27.57 -> 20.46 | Helped |
| use EquiformerV2 instead of eSCN | OC20 S2EF-All+MD · OC20 S2EF test | Force MAE: 15.6 -> 14.2 | Helped |
Surprising findings
- Simply incorporating eSCN convolutions into Equiformer with higher degrees does not beat the original eSCN baseline; the architectural changes are necessary.
- Directly replacing gate activation with S2 activation in Equiformer does not converge, while separable S2 activation trains stably.
Limitations
- Gains from higher degrees and architectural changes depend on dataset and task; small datasets like QM9 show limited improvement.
- Energy predictions benefit less than force predictions from improved expressivity.
- Equiformer cannot scale past L_max=2 on OC20 due to out-of-memory errors, so direct comparisons at higher degrees are limited.
Experimental setup
Datasets: OC20 S2EF-2M OC20 S2EF-All OC20 S2EF-All+MD OC22 S2EF-Total QM9 OC20 IS2RE
Benchmarks: OC20 S2EF OC20 IS2RS OC20 IS2RE OC22 S2EF-Total OC22 IS2RE-Total QM9 AdsorbML
Metrics: force MAE energy MAE AFbT ADwT success rate speedup throughput (samples per GPU second) training GPU-days
Baselines: eSCN GemNet-OC GemNet-dT SCN SchNet DimeNet++ SpinConv Equiformer PaiNN SEGNN
Builds on: Equiformer eSCN SCN gate activation Spherical CNNs S2 activation
Closest papers by meaning
- E2Former: An Efficient and Equivariant Transformer with Linear-Scaling Tensor Products (NeurIPS 2025 · similarity 0.90)
- Equivariant Spherical Transformer for Efficient Molecular Modeling (NeurIPS 2026 · similarity 0.87)
- Compact SO(3) Equivariant Atomistic Foundation Models via Structural Pruning (NeurIPS 2026 · similarity 0.86)
- ELoRA: Low-Rank Adaptation for Equivariant GNNs (ICML 2025 · similarity 0.86)
- Scalable Non-Equivariant 3D Molecule Generation via Rotational Alignment (ICML 2025 · similarity 0.85)
- Generalist Equivariant Transformer Towards 3D Molecular Interaction Learning (ICML 2024 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.