Research map / Large language models / Instruction and fine-tuning
Fine-Tuned Language Models Generate Stable Inorganic Materials as Text
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Fine-tunes LLaMA-2 models (7B/13B/70B) with LoRA on text-encoded crystal structures under multitask prompts, generating stable inorganic materials that outperform CDVAE on M3GNet/DFT stability.
Problem
Generating stable 3D inorganic crystal structures is hard because atoms mix discrete element identities with continuous coordinates and require symmetry handling; prior generative models are specialized diffusion or graph architectures.
Approach
Encode crystals as newline-separated text strings with LLaMA-2's digit-level tokenizer, combine with text prompts, and parameter-efficiently fine-tune a pretrained LLM (LoRA) with a multitask curriculum (generation, text-conditioning, infilling) and random translation augmentation, with no crystal-specific architecture.
Key results
- LLaMA-2 70B (temperature 0.7) reaches 49.8% metastable by M3GNet versus 28.8% for CDVAE.
- LLaMA-2 70B at temperature 0.7 achieves 99.6% structural validity and 95.4% compositional validity.
- DFT-verified stable rate is 10.6% for LLaMA-2 70B versus 5.4% for CDVAE.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Fine-tune LLaMA-2 70B rather than use CDVAE | Unconditional crystal generation · MP-20 | Metastable (M3GNet): 28.8% -> 49.8% | Helped |
| Fine-tune LLaMA-2 70B rather than use CDVAE | Unconditional crystal generation · MP-20 | Stable (DFT): 5.4% -> 10.6% | Helped |
| Lower sampling temperature from τ=1.0 to τ=0.7 (ablation) | LLaMA-2 70B · MP-20 | Metastable (M3GNet): 35.4% -> 49.8% | Helped |
| Lower sampling temperature from τ=1.0 to τ=0.7 (ablation) | LLaMA-2 13B · MP-20 | Metastable (M3GNet): 33.4% -> 38.0% | Helped |
| Lower sampling temperature from τ=1.0 to τ=0.7 (ablation) | LLaMA-2 7B · MP-20 | Metastable (M3GNet): 35.1% -> 35.0% | No clear effect |
| Use fractional rather than absolute coordinates (ablation) | LLaMA-2 7B · MP-20 | Structural validity: 90.8% -> 91.4% | Helped |
| Use fractional rather than absolute coordinates (ablation) | LLaMA-2 7B · MP-20 | Compositional validity: 80.5% -> 83.2% | Helped |
| Add atom-order permutations (ablation) | LLaMA-2 7B · MP-20 | Structural validity: 92.5% -> 89.2% | Hurt |
| Add atom-order permutations (ablation) | LLaMA-2 7B · MP-20 | Compositional validity: 82.9% -> 81.7% | Hurt |
| Use the shared LLaMA-2 tokenizer instead of special crystal tokens (ablation) | LLaMA-2 fine-tuning · Crystal training examples | Training loss (Epoch 5): 0.588 -> 0.385 | Helped |
| Guide element swaps with the infilling language model rather than uniform sampling | Template-based material mutation and relaxation · Materials sampled from the test set | Stable compounds | Helped |
| Increase model scale | LLaMA-2 trained on atomistic data · Translation symmetry evaluation | Increase in Perplexity under Transformation | Helped |
Surprising findings
- Training LLaMA-2 from random initialization gives much higher initial loss than pretrained initialization, showing text pretraining matters for LoRA fine-tuning.
- Larger models learn translation invariance better, but structural novelty decreases with model size while compositional novelty increases.
Limitations
- LLMs are sensitive to prompt and tokenization choices, and can hallucinate unphysical elements such as 'Ln'.
- Training the largest (70B) model is expensive for many compute budgets.
- Stability is estimated with the M3GNet surrogate, and DFT is only run on M3GNet-metastable candidates; text-conditioning on stability and space group is only moderately reliable.
Experimental setup
Datasets: MP-20 Materials Project
Benchmarks: MP-20
Metrics: structural validity compositional validity coverage recall coverage precision wdist density wdist num elements metastable rate (M3GNet) stable rate (DFT) IPT diversity
Baselines: CDVAE LM-CH LM-AC
Builds on: LoRA CDVAE M3GNet LLaMA-2
Closest papers by meaning
- LLM Meets Diffusion: A Hybrid Framework for Crystal Material Generation (NeurIPS 2025 · similarity 0.91)
- Generative Hierarchical Materials Search (NeurIPS 2024 · similarity 0.91)
- Scalable Diffusion for Materials Generation (ICLR 2024 · similarity 0.90)
- FlowLLM: Flow Matching for Material Generation with Large Language Models as Base Distributions (NeurIPS 2024 · similarity 0.90)
- Invariant Tokenization of Crystalline Materials for Language Model Enabled Generation (NeurIPS 2024 · similarity 0.89)
- Periodic Materials Generation using Text-Guided Joint Diffusion Model (ICLR 2025 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.