Research map / Reinforcement learning / Policy gradient and actor-critic

Beyond Squared Error: Exploring Loss Design for Enhanced Training of Generative Flow Networks

ICLR 2025 · Reinforcement learning › Policy gradient and actor-critic · approach: Regularized scalable deep value learning · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Shows that GFlowNet regression losses correspond to f-divergences and proposes zero-forcing/zero-avoiding losses (Linex(1), Linex(1/2), Shifted-Cosh) that improve convergence, diversity and robustness over the quadratic balance loss.

Problem

GFlowNet training almost always uses the quadratic (squared log-ratio) regression loss, and the choice of regression loss, which affects exploration and exploitation, has not been theoretically studied.

Approach

Establishes a correspondence between regression loss g and f-divergence D_f over minimal cuts (g(t)=f(e^t)-∫f(s)/s ds), so that zero-forcing and zero-avoiding properties of a divergence transfer to the loss; uses this to design Linex(1), Linex(1/2) and Shifted-Cosh losses.

Key results

• On bit-sequence generation, Linex(1) found all modes in 5/5 runs for TB, DB and STB with median 98.0k/10.8k/20.3k steps, versus quadratic loss succeeding in only 1/5 TB runs.
• Quadratic (zero-forcing) and Shifted-Cosh achieved higher Spearman correlation (about 0.81-0.82) than zero-avoiding Linex losses (about 0.74-0.77) in bit-sequence generation.
• On molecule generation, zero-forcing losses had higher average reward while zero-avoiding Linex losses had lower pairwise similarity (more diverse samples); Linex(1/2) showed the best top-k robustness.

Measured effects reported in the paper

Surprising findings

• Quadratic loss found modes fastest initially in bit-sequence generation but was overtaken before finding all modes.
• Shifted-Cosh failed to find all modes in DB runs (0/5), despite being both zero-forcing and zero-avoiding.

Limitations

• Theorem's exact correspondence requires specific sampling/resampling weights that the experiments only approximately satisfy.
• Results are mixed: Shifted-Cosh works for TB and STB but fails for DB; Linex losses have lower Spearman correlation than quadratic loss.

Experimental setup

Datasets: hyper-grid (synthetic) bit-sequence generation (synthetic) sEH molecule generation with pretrained proxy

Benchmarks: hyper-grid bit-sequence generation molecule generation (sEH)

Metrics: L1 error between P_T and P_R number of modes found Spearman correlation average reward pairwise similarity top-k reward

Baselines: Quadratic (squared) loss FM-GFN DB-GFN TB-GFN STB-GFN

Builds on: GFlowNets trajectory balance detailed balance sub-trajectory balance flow matching

Closest papers by meaning

• On Divergence Measures for Training GFlowNets (NeurIPS 2024 · similarity 0.93)
• Flow Factorization for Efficient Generative Flow Networks (AAAI 2025 · similarity 0.92)
• Revisiting Non-Acyclic GFlowNets in Discrete Environments (ICML 2025 · similarity 0.91)
• Pessimistic Backward Policy for GFlowNets (NeurIPS 2024 · similarity 0.91)
• Looking Backward: Retrospective Backward Synthesis for Goal-Conditioned GFlowNets (ICLR 2025 · similarity 0.91)
• Pre-Training and Fine-Tuning Generative Flow Networks (ICLR 2024 · similarity 0.90)