Research map / Probabilistic and scientific modeling / Molecules and proteins
All that structure matches does not glitter
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Critically audits carbon-24, perov-5 and MP-20 crystal-structure-prediction benchmarks, releasing deduplicated/polymorph-aware dataset splits and two new metrics (METRe and cRMSE) to replace the one-to-one match rate.
Problem
Benchmarks for generative crystal structure prediction (CSP) rely on datasets containing duplicate structures and polymorphs randomly split across train/test, and on the one-to-one match rate metric, which is ill-defined for polymorphic compositions and can mislead.
Approach
Deduplicate carbon-24 using dynamically varied StructureMatcher tolerances (keeping ~40% unique), group polymorphs within the same split, and evaluate with METRe (matching every reference structure against all generated ones) plus cRMSE, which penalizes unmatched structures with stol.
Key results
- Carbon-24 contains only about 40% unique structures (4250 of 10153) after deduplication with varied StructureMatcher tolerances.
- On perov-5-polymorph-split, OMatG-TrigODE/LinearODE reach cRMSE around 0.158-0.160 and METRe around 77%, with improved cRMSE across most models relative to the original perov-5 split.
- On MP-20-polymorph-split, OMatG-LinearODE achieves METRe 70.5%, RMSE 0.056 and cRMSE 0.187, outperforming DiffCSP and FlowMM on that split.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| replace the random perov-5 split with a polymorph-aware split | DiffCSP, FlowMM, and OMatG · perov-5 | cRMSE: 0.253 -> 0.162; 0.264 -> 0.161; 0.322 -> 0.158 | Helped |
| replace the random MP-20 split with a polymorph-aware split using published hyperparameters | DiffCSP · MP-20 | cRMSE: 0.244 -> 0.279 | Hurt |
| replace the random MP-20 split with a polymorph-aware split using published hyperparameters | FlowMM · MP-20 | cRMSE: 0.210 -> 0.226 | Hurt |
| use a polymorph-aware MP-20 split with hyperparameter tuning | OMatG-LinearODE · MP-20 | cRMSE: 0.208 -> 0.187 | Helped |
| add lattice angle and atom-count N information to CSPNet (ablation) | OMatG-LinearODE with CSPNet encoder · carbon-NXL (all N) | standard match rate: 47.3\,\% -> 47.7\,\% | Helped |
| add lattice angle and atom-count N information to CSPNet (ablation) | OMatG-LinearODE with CSPNet encoder · carbon-NXL (N=6) | standard match rate: 60.0\,\% -> 69.2\,\% | Helped |
| add lattice angle and atom-count N information to CSPNet (ablation) | OMatG-LinearODE with CSPNet encoder · carbon-NXL (N=8) | standard match rate: 39.0\,\% -> 26.0\,\% | Hurt |
| add lattice angle and atom-count N information to CSPNet (ablation) | OMatG-LinearODE with CSPNet encoder · carbon-X | standard match rate: 100.0\,\% -> 100.0\,\% | No clear effect |
| train on the high-to-low rather than low-to-high atom-count split | DiffCSP, FlowMM, and OMatG · carbon-24-unique-N-split | cRMSE: 0.429 -> 0.077; 0.406 -> 0.043; 0.402 -> 0.045 | Helped |
| use trigonometric rather than linear positional interpolation (ablation) | OMatG · carbon-24-unique | cRMSE: 0.189 -> 0.187 | Helped |
| use trigonometric rather than linear positional interpolation (ablation) | OMatG · perov-5 | cRMSE: 0.322 -> 0.381 | Hurt |
| increase unit-cell atom count from N=6 to N=8 | OMatG-LinearODE on carbon-NXL · carbon-NXL | standard match rate: 60.0\,\% -> 39.0\,\% | Hurt |
Surprising findings
- Standard StructureMatcher tolerances imply carbon-24 uniqueness of only about 3-4%, showing the match rate is highly tolerant.
- Models trained on the low-to-high N split perform poorly, showing extrapolation to larger unit cells is a major challenge.
Limitations
- Duplicate identification via StructureMatcher is only a likely-candidate method limited by numerical tolerances and the ambiguity of defining a match.
- METRe becomes inflated and harder to interpret if the test set contains many duplicates.
- Deduplicated carbon-24 test set may leak structures from the duplicated training set, since the split was not made to avoid overlap.
Experimental setup
Datasets: carbon-24 perov-5 MP-20 Alex-MP-20
Benchmarks: carbon-24-unique perov-5 perov-5-polymorph-split MP-20 MP-20-polymorph-split carbon-NXL carbon-X carbon-24-unique-N-split
Metrics: match rate METRe RMSE cRMSE k-match rate
Baselines: DiffCSP FlowMM OMatG-LinearODE OMatG-TrigODE k-match rate
Builds on: DiffCSP FlowMM OMatG CDVAE Pymatgen StructureMatcher
Closest papers by meaning
- Scalable Diffusion for Materials Generation (ICLR 2024 · similarity 0.91)
- WyckoffDiff -- A Generative Diffusion Model for Crystal Symmetry (ICML 2025 · similarity 0.90)
- LLM Meets Diffusion: A Hybrid Framework for Crystal Material Generation (NeurIPS 2025 · similarity 0.89)
- Generative Hierarchical Materials Search (NeurIPS 2024 · similarity 0.89)
- Open Materials Generation with Stochastic Interpolants (ICML 2025 · similarity 0.88)
- Fine-Tuned Language Models Generate Stable Inorganic Materials as Text (ICLR 2024 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.