Research map / Graph learning / Graph self-supervised learning
Learning-Order Autoregressive Models with Application to Molecular Graph Generation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Learning-Order ARMs (LO-ARM), which learn a state-dependent generation order via a variational ELBO with RLOO gradients, and reports state-of-the-art FCD on QM9 and ZINC250k molecular graph generation.
Problem
Autoregressive models need a generation order, but data like graphs and images lack a canonical ordering; fixed orders impose inductive bias and any-order models are hard to train and less effective in likelihood.
Approach
Replace the uniform permutation prior of AO-ARMs with a learnable, state-dependent order-policy, train it with a variational lower bound using an amortized Plackett-Luce posterior and RLOO REINFORCE gradients, and apply it to molecular graphs as unmasking of dense adjacency matrices and node vectors.
Key results
- On QM9, LO-ARM-st-sep reaches FCD 0.240 versus 0.441 for CatFlow (previous best) with 99.85% validity.
- On ZINC250k, LO-ARM-st-sep achieves FCD 3.229, improving over CatFlow's 13.21 and AO-ARM's 6.541, with 96.26% validity.
- LO-ARM-st-sep matches the best QED (0.75) on ZINC250k and attains similarity 0.34, while SAS is 3.08 vs 2.89 for Taiga.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Replace uniform ordering with a learned order-policy | AO-ARM on QM9 · QM9 generated samples | FCD: 0.671 -> 0.240 | Helped |
| Replace uniform ordering with a learned order-policy | AO-ARM on ZINC250k · ZINC250k generated samples | Validity (%): 32.93 -> 96.26 | Helped |
| Use edge-first ordering instead of uniform sampling across edges and nodes | AO-ARM on ZINC250k · ZINC250k generated samples | FCD: 6.541 -> 5.026 | Helped |
| Use a separate variational policy network rather than shared torso (ablation) | LO-ARM-st-st on QM9 · QM9 generated samples | FCD: 0.465 ± 0.027 -> 0.264 ± 0.031 | Helped |
| Use entropy-based order policy rather than shared-torso model policy (ablation) | LO-ARM-st-st on QM9 · QM9 test | NLL: ≤22.23 ± 0.01 -> ≤24.05 ± 0.29 | Hurt |
| Use entropy-based order policy rather than shared-torso model policy (ablation) | LO-ARM-st-sep on QM9 · QM9 generated samples | FCD: 0.264 ± 0.031 -> 0.444 ± 0.013 | Hurt |
| Add Top-p sampling with p=0.9 (ablation) | LO-ARM-st-st on ZINC250k · ZINC250k generated samples | Validity (%): 90.84 -> 96.02 | Helped |
| Add Top-p sampling with p=0.9 (ablation) | LO-ARM-st-st on ZINC250k · ZINC250k generated samples | FCD: 4.087 -> 9.237 | Hurt |
| Add Top-p sampling with p=0.3 (ablation) | LO-ARM-st-sep on ZINC250k · ZINC250k generated samples | FCD: 3.229 -> 3.859 | Hurt |
| Add Top-p sampling with p=0.3 (ablation) | LO-ARM-st-sep on ZINC250k · ZINC250k generated samples | Validity (%): 96.26 -> 96.70 | Helped |
Surprising findings
- Without inductive bias, the learned order is edge-first in 99.9% of 30,000 sampled molecules across three seeds.
- ZINC250k validity remains lower than diffusion-based methods such as CatFlow (96.26% vs 99.21%), despite much better FCD.
Limitations
- Validity on ZINC250k lags diffusion-based methods, attributed to error accumulation during autoregressive sampling.
- Dense adjacency matrix representation is memory-inefficient for larger graphs.
- Top-p sampling improves validity but degrades FCD.
Experimental setup
Datasets: QM9 ZINC250k MNIST
Benchmarks: QM9 ZINC250k
Metrics: NLL validity uniqueness FCD QED SAS similarity
Baselines: AO-ARM Biased-AO-ARM GDSS DiGress GraphARM CatFlow GraphDF MoFlow SPECTRE EDP-GNN GraphAF JT-VAE
Builds on: AO-ARM DiGress CatFlow RLOO
Closest papers by meaning
- NEAT: Neighborhood-Guided, Efficient, Autoregressive Set Transformer for 3D Molecular Generation (NeurIPS 2026 · similarity 0.87)
- Pard: Permutation-Invariant Autoregressive Diffusion for Graph Generation (NeurIPS 2024 · similarity 0.86)
- Aligning Transformers with Continuous Feedback via Energy Rank Alignment (NeurIPS 2025 · similarity 0.85)
- BARNN: A Bayesian Autoregressive and Recurrent Neural Network (ICML 2025 · similarity 0.85)
- SNACK: A Sequential Notation Framework for Probabilistic Graph Generation (NeurIPS 2026 · similarity 0.84)
- Curriculum Learning for Biological Sequence Prediction: The Case of De Novo Peptide Sequencing (ICML 2025 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.