Research map / Reinforcement learning / Policy gradient and actor-critic
Looking Backward: Retrospective Backward Synthesis for Goal-Conditioned GFlowNets
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Retrospective Backward Synthesis (RBS), which synthesizes backward trajectories from goals using the learned backward policy to densify sparse rewards in goal-conditioned GFlowNets, and reports large sample-efficiency gains on GridWorld, bit sequence, TF Bind and AMP tasks.
Problem
Goal-conditioned GFlowNets receive extremely sparse binary rewards in high-dimensional tasks and have limited trajectory coverage, which is especially problematic in offline settings.
Approach
Augment forward trajectories with newly synthesized backward trajectories sampled from the goal via the learned backward policy (which always reach the goal, so they carry positive reward), combined with age-based replay sampling, backward policy KL regularization toward uniform, and an intensified terminal reward coefficient C.
Key results
- On large-scale sequence generation, RBS-GFN reaches about 100% success rate while all baselines completely fail, per the paper's claim.
- On unseen-obstacle GridWorld testing, RBS-GFN maintains almost 100% success while OC-GAFN obtains 9% and DQN completely fails.
- Without intensified reward feedback, RBS-GFN obtains 0% success on bit sequence generation tasks.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add retrospective backward synthesis of goal-reaching trajectories | GC-GFlowNet · GridWorld | success rate | Helped |
| Add retrospective backward synthesis of goal-reaching trajectories | GC-GFlowNet · GridWorld (unseen goals) | success rate: approximately 15% | Helped |
| Add retrospective backward synthesis of goal-reaching trajectories | GC-GFlowNet · Bit sequence generation | success rate: approximately 100% | Helped |
| Add retrospective backward synthesis | GC-GFlowNet · TF Bind generation | success rate: 100% | Helped |
| Add hierarchical goal decomposition | RBS-GFN · AMP generation | learning efficiency | Helped |
| Add age-based sampling of new experiences (ablation) | RBS-GFN · GridWorld (H=128) | learning efficiency | Helped |
| Add age-based sampling of new experiences (ablation) | RBS-GFN · Bit sequence generation (medium and large) | generalization | Helped |
| Add backward-policy regularization toward a uniform distribution (ablation) | RBS-GFN · GridWorld | success rate | Helped |
| Use a fixed uniform backward policy instead of a regularized learned backward policy (ablation) | RBS-GFN · GridWorld | success rate | Hurt |
| Use an unconstrained learned backward policy instead of a regularized learned backward policy (ablation) | RBS-GFN · GridWorld | success rate | Hurt |
| Remove intensified reward feedback (ablation) | RBS-GFN · Bit sequence generation | success rate: 0% | Hurt |
| Replace age-based sampling with prioritized experience replay (ablation) | RBS-GFN · GridWorld | learning efficiency | Hurt |
Surprising findings
- SAC with HER performs worse than DQN with HER on TF Bind, which the authors attribute to entropy maximization favoring novel states over goal completion.
- Age-based sampling outperforms prioritized experience replay (PER) by a large margin, which the authors attribute to unstable GC-GFlowNet loss causing PER to restrict coverage.
Limitations
- Focuses on deterministic, discrete environments with known dynamics; continuous environments left for future work.
- Intensification coefficient C requires task-dependent tuning (e.g., 1e7 to 1e40), and the method's success depends on it.
- Extremely large state spaces (e.g., AMP with 20^50 states) need a hierarchical goal decomposition extension that is manually designed.
Experimental setup
Datasets: GridWorld (H=32, 64, 128) bit sequence generation TF Bind AMP (antimicrobial peptides) RNA generation offline GridWorld dataset of 8,016 trajectories
Benchmarks: GridWorld bit sequence generation TF Bind generation AMP generation unseen-goal and unseen-obstacle GridWorld generalization downstream finetuning bit sequence
Metrics: success rate entropy of forward policy number of modes discovered
Baselines: GFN w/ HER OC-GAFN DQN w/ HER SAC w/ HER vanilla GC-GFlowNet MHER DreamerV3-based GC-RL PER variant
Builds on: GFlowNets trajectory balance sub-trajectory balance hindsight experience replay OC-GAFN
Closest papers by meaning
- Optimizing Backward Policies in GFlowNets via Trajectory Likelihood Maximization (ICLR 2025 · similarity 0.94)
- Pessimistic Backward Policy for GFlowNets (NeurIPS 2024 · similarity 0.94)
- Pre-Training and Fine-Tuning Generative Flow Networks (ICLR 2024 · similarity 0.93)
- GFlowNet Training by Policy Gradients (ICML 2024 · similarity 0.93)
- Order-Preserving GFlowNets (ICLR 2024 · similarity 0.91)
- Flow Factorization for Efficient Generative Flow Networks (AAAI 2025 · similarity 0.91)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.