Research map / Architectures and efficiency / Attention and transformers
Crystalformer: Infinitely Connected Attention for Periodic Structure Encoding
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Crystalformer, a normalization-free Transformer encoder for crystal structures whose attention is an infinite distance-decay potential summation over periodic images, tractably approximated as pseudo-finite periodic attention.
Problem
Encoding periodic crystal structures with Transformers requires fully connected attention over infinitely repeated atoms (infinitely connected attention), which standard Transformers cannot handle directly; existing crystal Transformers (Matformer) depart from the standard architecture and existing
Approach
Interpret infinitely connected attention as a neural potential summation; use Gaussian distance-decay attention so the infinite image sums (periodic spatial and edge encodings) converge and can be computed approximately, plus value position encoding to distinguish lattices of the same monatomic cell; the architecture stays close to the original Transformer with no layer normalization.
Key results
- On Materials Project, Crystalformer reaches formation energy MAE 0.0186 and bandgap MAE 0.198, beating Matformer (0.021, 0.211) and second only to PotNet on some tasks.
- Crystalformer uses 853K parameters versus 2.9M for Matformer (29.4%) and 1.8M for PotNet, with the fastest inference (~6.6 ms per material).
- On JARVIS-DFT, Crystalformer is competitive with PotNet (e.g. formation energy 0.0306 vs 0.0294; E hull 0.0463 vs 0.055, best on E hull).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add value position encoding ψ (ablation) | Crystalformer · JARVIS-DFT 3D 2021, formation energy | MAE: 0.0541 -> 0.0301 | Helped |
| add value position encoding ψ (ablation) | Crystalformer · JARVIS-DFT 3D 2021, total energy | MAE: 0.0546 -> 0.0314 | Helped |
| add value position encoding ψ (ablation) | Crystalformer · JARVIS-DFT 3D 2021, bandgap OPT | MAE: 0.140 -> 0.133 | Helped |
| add value position encoding ψ (ablation) | Crystalformer · JARVIS-DFT 3D 2021, bandgap MBJ | MAE: 0.308 -> 0.287 | Helped |
| add value position encoding ψ (ablation) | Crystalformer · JARVIS-DFT 3D 2021, E hull | MAE: 0.0517 -> 0.0487 | Helped |
| use stochastic weight averaging (SWA) (ablation) | Crystalformer · Materials Project (MEGNet), formation energy | MAE: 0.0198 -> 0.0186 | Helped |
| use stochastic weight averaging (SWA) (ablation) | Crystalformer · Materials Project (MEGNet), bandgap | MAE: 0.201 -> 0.198 | Helped |
| use stochastic weight averaging (SWA) (ablation) | Crystalformer · Materials Project (MEGNet), bulk modulus | MAE: 0.0399 -> 0.0377 | Helped |
| use stochastic weight averaging (SWA) (ablation) | Crystalformer · Materials Project (MEGNet), shear modulus | MAE: 0.0692 -> 0.0689 | Helped |
| use stochastic weight averaging (SWA) (ablation) | Crystalformer · JARVIS-DFT 3D 2021, formation energy | MAE: 0.0319 -> 0.0306 | Helped |
| use stochastic weight averaging (SWA) (ablation) | Crystalformer · JARVIS-DFT 3D 2021, total energy | MAE: 0.0342 -> 0.0320 | Helped |
| use stochastic weight averaging (SWA) (ablation) | Crystalformer · JARVIS-DFT 3D 2021, bandgap OPT | MAE: 0.131 -> 0.128 | Helped |
Surprising findings
- Removing value position encoding (psi/beta) costs most training time, and the ablation shows the largest gains on formation and total energy but smaller on other properties.
Limitations
- Distance-only (Gaussian) position encodings limit expressiveness and ignore angular/directional information.
- Gaussian decay upper bound on sigma restricts modeling of long-range interactions such as Coulomb; a reciprocal-space variant is only tentatively evaluated.
- Total training time is longer than baselines (7.2 h vs 5.9 h for PotNet on formation energy).
Experimental setup
Datasets: Materials Project (MEGNet) JARVIS-DFT 3D 2021
Benchmarks: Materials Project (MEGNet) JARVIS-DFT 3D 2021
Metrics: MAE
Baselines: CGCNN SchNet MEGNet GATGNN M3GNet ALIGNN Matformer PotNet
Builds on: Graphormer Matformer Transformer CGCNN
Closest papers by meaning
- Complete and Efficient Graph Transformers for Crystal Material Property Prediction (ICLR 2024 · similarity 0.90)
- Conformal Crystal Graph Transformer with Robust Encoding of Periodic Invariance (AAAI 2024 · similarity 0.89)
- Learning the Electronic Hamiltonian of Large Atomic Structures (ICML 2025 · similarity 0.89)
- A Diffusion-Based Pre-training Framework for Crystal Property Prediction (AAAI 2024 · similarity 0.87)
- Vector Field Oriented Diffusion Model for Crystal Material Generation (AAAI 2024 · similarity 0.87)
- A Denoising Pre-training Framework for Accelerating Novel Material Discovery (AAAI 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.