Research map / Reinforcement learning / Policy gradient and actor-critic
Order-Preserving GFlowNets
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Order-Preserving GFlowNets (OP-GFNs), which learn a reward consistent with a (partial) order over candidates instead of an explicit scalar reward, applied to single-objective maximization and multi-objective Pareto front approximation.
Problem
GFlowNets need an explicit scalar reward, which is hard to define for multi-objective optimization and costly to evaluate in NAS; standard reward exponent beta for prioritizing high-reward candidates is environment-dependent.
Approach
Train the GFlowNet to sample proportionally to a learned reward that matches the ordering of candidates via a KL loss between Pareto-based labeling and the reward-induced distribution over batches, so only pairwise/partial order information is needed; the learned reward sparsifies over training, balancing exploration and exploitation.
Key results
- On NAS (NATS-Bench), OP-TB variants consistently improve over TB, REA, BOHB and REINFORCE in best test accuracy at epochs 12 and 200, especially in early training.
- OP-TB outperforms GFN-beta (beta=4 to 128) by a significant margin on NAS.
- Multi-objective experiments report OP-GFNs achieve better or comparable Pareto front approximation to PC-GFN/GC-GFN on HyperGrid, DNA, and fragment-based molecule tasks.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add order-preserving loss (OP-TB) to trajectory balance | TB on HyperGrid · HyperGrid (D,H)=(2,64) | exploration-exploitation ratios (distinct visited maximal states, maximal candid | Helped |
| Add order-preserving loss to TB | TB on NATS-Bench topology search · NATS-Bench CIFAR10/100/ImageNet-16-120 | best test accuracy at epochs 12 and 200 vs. T&T time | Helped |
| Add backward KL regularization and trajectory augmentation to OP-TB (ablation) | OP-TB on NATS-Bench · NATS-Bench | sampling efficiency (accuracy vs T&T time) | Helped |
| Add order-preserving loss to non-TB objectives (FM, DB, subTB) (ablation) | FM, DB, subTB on NATS-Bench · NATS-Bench | accuracy vs T&T time | Helped |
| Replace GFN-beta reward exponent with order-preserving reward (OP-TB) | GFN-beta on NATS-Bench · NATS-Bench | accuracy vs T&T time | Helped |
| Add order-preserving loss to TB on molecular design (Bag, QM9, sEH, TFBind) | TB on molecule tasks · Molecular design environments | top-100 average reward and number of optimal candidates found | Helped |
| Use order-preserving reward for multi-objective sampling (OP-GFN vs preference-conditioning PC-GFN) on HyperGrid | multi-objective HyperGrid (D,H)=(2,32) · HyperGrid 4 objectives | IGD+, hypervolume, PC-entropy | Helped |
| Use OP-GFN instead of PC-GFN on DNA sequence generation | DNA aptamer generation (3 objectives) · DNA sequence generation | Pareto front metrics and diversity | Mixed |
| Use OP-GFN instead of PC/GC-GFN on fragment-based molecule generation | 4-objective molecule generation · fragment-based molecule generation | Pareto front quality | No clear effect |
| Use OP-GFN instead of baselines on n-gram multi-objective task | n-gram synthetic sequence design · n-gram | Pareto metrics | Helped |
Surprising findings
- The learned order-preserving reward is theoretically shown to become sparser during training, which the authors argue naturally shifts the sampler from exploration to exploitation without tuning beta.
Limitations
- Diversity of multi-objective sampling depends on replay buffer resampling; collapse to part of the Pareto front is possible.
- Theoretical results are for the single-objective case with specific assumptions (pairwise labels, uniform backward policy in the MDP proposition).
- Non-TB parametrizations need a tuned hyperparameter lambda_OP.
Experimental setup
Datasets: HyperGrid NATS-Bench Bag TFBind8 TFBind10 QM9 sEH n-gram DNA aptamer sequences (NUPACK)
Benchmarks: HyperGrid NATS-Bench Bag TFBind8 TFBind10 QM9 sEH n-gram DNA sequence generation fragment-based molecule generation
Metrics: top-100 mean reward number of optimal candidates found best test accuracy at epochs 12 and 200 IGD+ hypervolume R2 indicator PC-entropy averaged Hausdorff distance
Baselines: TB DB subTB MaxEnt GTB MARS A2C SQL PPO REA REINFORCE BOHB
Builds on: GFlowNet trajectory balance detailed balance preference-conditioning GFlowNets goal-conditioning GFlowNets
Closest papers by meaning
- Pre-Training and Fine-Tuning Generative Flow Networks (ICLR 2024 · similarity 0.92)
- Looking Backward: Retrospective Backward Synthesis for Goal-Conditioned GFlowNets (ICLR 2025 · similarity 0.91)
- Flow Factorization for Efficient Generative Flow Networks (AAAI 2025 · similarity 0.91)
- Pessimistic Backward Policy for GFlowNets (NeurIPS 2024 · similarity 0.91)
- Optimizing Backward Policies in GFlowNets via Trajectory Likelihood Maximization (ICLR 2025 · similarity 0.90)
- GFlowNet Training by Policy Gradients (ICML 2024 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.