Research map / Reinforcement learning / Policy gradient and actor-critic
QGFN: Controllable Greediness with Action Values
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes QGFN, which mixes a GFlowNet forward policy with a jointly trained n-step action-value function (p-greedy, p-quantile, p-of-max variants) to get controllably greedy yet diverse sampling at inference.
Problem
GFlowNets sample proportionally to reward, so producing consistently high-reward samples requires tuning an inverse temperature beta, which can cause diversity collapse and numerical instability; greediness is hard to control.
Approach
Combine the GFN forward policy P_F with an action-value estimate Q trained off-policy on the same minibatches (n-step TD), forming greedier behavior policies (mixture, quantile masking, or fraction-of-max masking) whose greediness parameter p can be changed at inference without retraining.
Key results
- On the fragment task, p-of-max QGFN reaches reward 0.969±0.003 with similarity 0.514, versus 0.780 for GFN-TB and 0.716 for GFN-SubTB at inference.
- QGFN variants discover more modes than GFN baselines across tasks while keeping inter-sample similarity low.
- Changing p at inference on a p-greedy-trained model raises average reward without retraining, and Q-guided inference on independently trained TB/SubTB models improves their reward (e.g., SubTB 0.716 to 0.939 with p-of-max).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Combine the GFN forward policy with Q-guided action selection | GFN sampling · Fragment-based molecule generation | Reward: 0.780$\pm$0.003 -> 0.950$\pm$0.004; 0.780$\pm$0.003 -> \textbf{0.969$\pm$0.003}; 0.780$\pm$0.003 -> 0.955$\pm$0.003 | Helped |
| Use p-of-max QGFN instead of GFN-TB | GFN-TB · Fragment-based molecule generation, inference | Similarity: 0.545$\pm$0.002 -> 0.514$\pm$0.001 | No clear effect |
| Use p-quantile QGFN instead of GFN-TB | GFN-TB · Fragment-based molecule generation, inference | Reward: 0.780$\pm$0.003 -> 0.955$\pm$0.003 | Helped |
| Apply trained Q-guided inference to an independently trained TB model | TB · Fragment-based molecule generation | Reward: 0.780$\pm$0.003 -> 0.936$\pm$0.009 | Helped |
| Apply trained Q-guided inference to an independently trained SubTB model | SubTB · Fragment-based molecule generation | Reward: 0.716$\pm$0.006 -> 0.921$\pm$0.008; 0.716$\pm$0.006 -> \textbf{0.939$\pm$0.003}; 0.716$\pm$0.006 -> 0.911$\pm$0.009 | Helped |
| Use p-quantile pruning in a task with a small action space | QGFN · RNA-binding tasks | Performance | Hurt |
| Use p-quantile QGFN on the expanded bit-sequence action space | QGFN · Bit sequence task, expanded action space | Performance | Helped |
| Increase the Q-learning return from 1-step to n-step (ablation) | QGFN · Fragment-based molecule generation | Number of modes | Helped |
| Use a small n-step return (ablation) | QGFN · Fragment-based molecule generation | Number of modes | Hurt |
| Train with a different p value (ablation) | QGFN · Fragment-based molecule generation | Diversity | Mixed |
| Increase reward temperature beta (ablation) | QGFN and GFN · Fragment-based molecule generation | Reward | Helped |
| Increase reward temperature beta (ablation) | QGFN and GFN · Fragment-based molecule generation | Number of modes | Hurt |
Surprising findings
- Naively summing the GFlowNet and Q-learning losses via weight sharing slowed learning rather than helping.
- A DQN trained alongside the GFN for shared data slowed the whole method, a predecessor approach that failed to beat baselines.
Limitations
- Requires training two models, increasing memory, FLOPs and training time compared to TB.
- Performance is sensitive to the quality of Q, and n-step returns are crucial for it to work.
- Scaling behaviour with parameters and compute was not explored.
Experimental setup
Datasets: QM9
Benchmarks: fragment-based molecule (sEH) QM9 HOMO-LUMO gap L14-RNA1 L14-RNA1+2 prepend-append bit sequence
Metrics: average reward number of modes pairwise Tanimoto similarity
Baselines: GFN-TB GFN-SubTB LSL-GFN A2C DQN MunDQN
Builds on: GFlowNet Trajectory Balance Sub-Trajectory Balance Q-learning MunDQN
Closest papers by meaning
- Flow Factorization for Efficient Generative Flow Networks (AAAI 2025 · similarity 0.91)
- GFlowNet Training by Policy Gradients (ICML 2024 · similarity 0.90)
- Pre-Training and Fine-Tuning Generative Flow Networks (ICLR 2024 · similarity 0.90)
- Random Policy Evaluation Uncovers Policies of Generative Flow Networks (ICML 2025 · similarity 0.89)
- Pessimistic Backward Policy for GFlowNets (NeurIPS 2024 · similarity 0.89)
- Optimizing Backward Policies in GFlowNets via Trajectory Likelihood Maximization (ICLR 2025 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.