Research map / Reinforcement learning / RL from feedback
floq: Training Critics via Flow-Matching for Scaling Compute in Value-Based RL
Bhavya Kumar Agrawalla, Michal Nauman, Khush Agrawal, Aviral Kumar
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| replace monolithic Q-network with flow-matching Q-function | FQL · OGBench 50 tasks | average success rate (%): 46 -> 56 | Improved |
| tune flow steps and kappa per environment (Best vs Def.) | floq · OGBench 50 tasks | average success rate (%): 56 -> 59 | Improved |
| train FQL for 2M instead of 1M steps | FQL · OGBench 50 tasks | average success rate (%): 46 -> 47 | No clear effect |
| use flow-matching critic instead of distributional IQN critic | FQL · OGBench default tasks | average success rate (%): 49 -> 64 | Improved |
| add dense supervision over all t with multi-step integration (vs t=0 only) (ablation) | floq · OGBench antmaze-giant | success rate (%): 39 -> 86 | Improved |
| add dense supervision over all t with multi-step integration (vs t=0 only) (ablation) | floq · OGBench humanoidmaze-large | success rate (%): 49 -> 56 | Improved |
| add dense supervision over all t with multi-step integration (vs t=0 only) (ablation) | floq · OGBench cube-double | success rate (%): 38 -> 72 | Improved |
| add dense supervision over all t with multi-step integration (vs t=0 only) (ablation) | floq · OGBench humanoidmaze-medium | success rate (%): 94 -> 98 | Improved |
| increase ensemble of monolithic critics to 8 (compute matched to 8 flow steps) | FQL · OGBench | success rate | Hurt |
| use HL-Gauss embedding of interpolant instead of scalar or normalized scalar (ablation) | floq · OGBench | success rate | Improved |
| increase HL-Gauss sigma from {1,2,4,8} to 16 (ablation) | floq · OGBench | success rate | Improved |
| use 64-dim Fourier time embedding instead of scalar t (ablation) | floq · OGBench | success rate | Improved |
Closest papers by meaning
- Flow Q-Learning (ICML 2025 · similarity 0.89)
- Stop Regressing: Training Value Functions via Classification for Scalable Deep RL (ICML 2024 · similarity 0.87)
- Long-Horizon Q-Learning: Accurate Value Learning via n-Step Inequalities (NeurIPS 2026 · similarity 0.87)
- Compute-Optimal Scaling for Value-Based Deep RL (NeurIPS 2025 · similarity 0.87)
- Q-SFT: Q-Learning for Language Models via Supervised Fine-Tuning (ICLR 2025 · similarity 0.87)
- Online Reward-Weighted Fine-Tuning of Flow Matching with Wasserstein Regularization (ICLR 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
