Research map / Large language models / Language, safety and interpretability
FlowLLM: Flow Matching for Material Generation with Large Language Models as Base Distributions
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes FlowLLM, a hybrid that samples crystal structures from a LoRA-fine-tuned LLaMA-2 70B and refines fractional coordinates and lattice parameters with a Riemannian flow matching model, reaching about 3x higher stability rate than prior methods on MP-20.
Problem
Generating stable, unique, novel crystalline materials requires jointly handling discrete atom types and continuous coordinates and lattice geometry, which existing LLM-based and denoising-based generators each handle only partially.
Approach
Use a fine-tuned LLM's output distribution as the learned base distribution for Riemannian flow matching, so the flow only needs to refine coordinates and lattice parameters from an already chemically plausible starting structure, rather than from a uniform or Gaussian base.
Key results
- Best FlowLLM setting (tau=0.7, P=0.9, 250 steps) reaches 17.82% stability rate vs 4.65% for FlowMM and 5.28% for CrystalLLM (70B) on MP-20.
- S.U.N. rate of 4.92% for the best FlowLLM setting vs 3.34% for DiffCSP and 2.34% for FlowMM.
- Relaxed-structure comparison: match rate 94.9% vs 74.3% for FlowMM, RMSD 0.023 Å vs 0.096 Å, and 37.97 vs 191.98 relaxation steps.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use learned LLM base distributions for lattice parameters and fractional coordinates instead of simple base distributions (ablation) | FlowLLM · MP-20 | Stability Rate (%): 8.95 -> 17.82 | Helped |
| Use learned LLM base distributions for lattice parameters and fractional coordinates instead of simple base distributions (ablation) | FlowLLM · MP-20 | S.U.N. Rate (%) | Helped |
| Add Gaussian noise with std 0.01 to continuous LLM outputs (ablation) | FlowLLM · MP-20 generation | Structural Validity (%): 99.64 -> 99.85 | No clear effect |
| Add Gaussian noise with std 0.02 to continuous LLM outputs (ablation) | FlowLLM · MP-20 generation | Structural Validity (%): 99.64 -> 99.99 | No clear effect |
| Add Gaussian noise with std 0.04 to continuous LLM outputs (ablation) | FlowLLM · MP-20 generation | Structural Validity (%): 99.64 -> 99.97 | No clear effect |
| Use FlowLLM instead of FlowMM | Material generation and relaxation · Generated vs CHGNet-relaxed structures | Match Rate (%): 74.3 -> 94.9 | Helped |
| Use FlowLLM instead of FlowMM | Material generation and relaxation · Generated vs CHGNet-relaxed structures | RMSD (Å): 0.096 -> 0.023 | Helped |
| Use FlowLLM instead of FlowMM | Material generation and relaxation · Generated vs CHGNet-relaxed structures | Δ-Energy (eV/atom): 0.3031 -> 0.0898 | Helped |
| Use FlowLLM instead of FlowMM | Material generation and relaxation · Generated vs CHGNet-relaxed structures | Num Steps: 191.98 -> 37.97 | Helped |
| Use FlowLLM instead of the strongest prior model | MP-20 material generation · MP-20 | Stability Rate (%): 5.28 -> 17.82 | Helped |
| Use FlowLLM instead of FlowMM | MP-20 generation · MP-20 | Time to generate a S.U.N. material (seconds): 16.14 -> 10.9 | Helped |
Surprising findings
- FlowLLM using only an LLM for atom types with the standard FlowMM base distributions still beats prior models on stability, but is much worse than the full learned-base version, showing the value of the LLM base for coordinates and lattices.
- Adding Gaussian noise to LLM outputs made no noticeable difference to proxy metrics, despite the LLM base being discrete.
Limitations
- Lack of end-to-end differentiability hinders inverse design.
- LLM base distribution is not exactly invariant to permutation or translation symmetries.
- Many proxy metrics are saturated and offer limited discrimination.
Experimental setup
Datasets: MP-20 Materials Project
Benchmarks: MP-20
Metrics: stability rate S.U.N. rate structural validity composition validity coverage recall coverage precision wasserstein distance of density wasserstein distance of N_el match rate RMSD
Baselines: CDVAE DiffCSP FlowMM CrystalLLM
Builds on: FlowMM CrystalLLM Riemannian flow matching DiffCSP
Closest papers by meaning
- FlowMM: Generating Materials with Riemannian Flow Matching (ICML 2024 · similarity 0.93)
- Fine-Tuned Language Models Generate Stable Inorganic Materials as Text (ICLR 2024 · similarity 0.90)
- LLM Meets Diffusion: A Hybrid Framework for Crystal Material Generation (NeurIPS 2025 · similarity 0.90)
- Scalable Diffusion for Materials Generation (ICLR 2024 · similarity 0.90)
- Generative Hierarchical Materials Search (NeurIPS 2024 · similarity 0.89)
- Open Materials Generation with Stochastic Interpolants (ICML 2025 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.