Research map / Graph learning / Graph self-supervised learning
Graph Diffusion Policy Optimization
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes GDPO, a policy-gradient method that fine-tunes discrete graph diffusion models (DiGress) on arbitrary reward signals using an "eager policy gradient" that weights x0-prediction log-likelihoods instead of per-step transition log-likelihoods.
Problem
Optimizing discrete graph diffusion models for non-differentiable, arbitrary downstream objectives (e.g., molecular docking, validity, distribution similarity) where direct REINFORCE-style policy gradients (DDPO) fail to converge due to high variance on discrete graph trajectories.
Approach
Formulate graph DPM denoising as an MDP and replace the per-step transition log-likelihood gradient in REINFORCE with the gradient of log p(G0|Gt) at all timesteps ("eager policy gradient"), a biased but lower-variance estimator that directly promotes predicting high-reward G0 from every noisy state.
Key results
- On Planar, GDPO reduces Deg/Clus/Orb distances by an average 81.97% relative to the best baseline DiGress.
- On ZINC250k, GDPO achieves Hit Ratio of 34.359 on 5ht1b versus 18.673 for MOOD, and averages a 5.72% Hit Ratio improvement over other SOTA methods.
- DDPO (REINFORCE on graph DPMs) fails to converge for graphs with more than 4 nodes in a toy connectivity experiment, while GDPO performs consistently well.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| replace one-step denoising gradients with eager gradients on the predicted clean graph | Graph DPM optimization · Toy connected-graph generation | average reward | Helped |
| replace DDPO with GDPO | Graph DPM optimization · Planar and SBM graphs | average distribution distance reduction vs DiGress: 81.97%; 41.64% | Helped |
| replace DDPO with GDPO | Molecular graph optimization · ZINC250k | Hit Ratio: 5.72% | Helped |
| replace DDPO with GDPO | Molecular graph optimization · ZINC250k | DS (top 5%): 1.48% | Helped |
| replace DDPO with GDPO | Molecular graph optimization · MOSES, four targets excluding 5ht1b | Hit Ratio: 12.94% | Helped |
| replace DDPO with GDPO | Molecular graph optimization · MOSES | DS (top 5%): 5.54% | Helped |
| increase the validity reward weight from 0.7 to 0.9 (ablation) | GDPO · Planar Graphs | Clus: 0.62 ± 0.11 -> 2.17 ± 0.84 | Hurt |
| increase the validity reward weight from 0.7 to 0.9 (ablation) | GDPO · Planar Graphs | V.U.N (%): 73.83± 2.49 -> 81.26 ± 3.02 | Helped |
| add importance sampling to DDPO (ablation) | DDPO · ZINC250k, 5ht1b | Hit Ratio: 5.488 ± 1.989 -> 10.304 ± 1.277 | Helped |
| add importance sampling to GDPO (ablation) | GDPO · ZINC250k | Hit Ratio: 9.814 ± 1.352 -> 0.850 ± 0.602 | Hurt |
| reduce the number of sampled trajectories (ablation) | GDPO · ZINC250k, 5ht1b | training stability | Hurt |
| assign a high weight to the novelty reward (ablation) | GDPO · Molecule property optimization | reward | Hurt |
Surprising findings
- Importance sampling improves DDPO but does not improve GDPO (GDPO-IS performs no better than GDPO), which the authors attribute to the bias of the eager gradient.
- Assigning a large weight to the novelty reward causes the model to rapidly degenerate, a failure mode the authors report.
Limitations
- Risk of overoptimization and distribution collapse when reward weights are high (e.g., novelty).
- Scalability to large graphs (500+ nodes) is unexplored.
- The eager policy gradient is a biased estimator of the true policy gradient, and its theoretical gap to DDPO is not analyzed.
Experimental setup
Datasets: ZINC250k MOSES SBM Planar
Benchmarks: SBM Planar ZINC250k MOSES
Metrics: Deg Clus Orb V.U.N (%) Hit Ratio DS (top 5%) Spectral MMD IoU Uniqueness
Baselines: GraphRNN SPECTRE GDSS MOOD DiGress DDPO GCPN REINVENT FREED DiGress-guidance
Builds on: DDPO DiGress REINFORCE DPOK
Closest papers by meaning
- Diffusion Policy Policy Optimization (ICLR 2025 · similarity 0.86)
- Diffuse, Sample, Project: Plug-And-Play Controllable Graph Generation (ICML 2024 · similarity 0.86)
- Discrete-state Continuous-time Diffusion for Graph Generation (NeurIPS 2024 · similarity 0.86)
- Gradient Guidance for Diffusion Models: An Optimization Perspective (NeurIPS 2024 · similarity 0.85)
- Fine-Tuning Discrete Diffusion Models with Policy Gradient Methods (NeurIPS 2025 · similarity 0.85)
- Training Diffusion Models with Reinforcement Learning (ICLR 2024 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.