Research map / Reinforcement learning / Offline RL

Trust Region Q Adjoint Matching

NeurIPS 2026 · Reinforcement learning › Offline RL · approach: Offline RL sample complexity theory · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes TRQAM, which adapts a trust-region parameter λ inside the memoryless SOC sampling process of Q Adjoint Matching via projected dual descent to enforce a target path-space KL bound, improving offline and offline-to-online RL on 50 OGBench tasks.

Problem

Off-policy fine-tuning of pretrained flow policies is unstable because imperfect learned critics, amplified by fixed-temperature critic-guided adjoint matching, cause large policy deviations and collapse.

Approach

Scale the control term in the memoryless SOC SDE by a trust-region parameter λ, which by Girsanov's theorem makes λ the exact weight of path-space KL in the SOC objective; adapt λ by projected dual descent to track a prescribed KL budget, instead of adding a loss-level KL penalty.

Key results

• TRQAM achieves 68% aggregate offline RL success across 50 OGBench tasks, versus 46% for DSRL, 45% for QAM-E and 35% for QAM.
• On Robomimic, fixed-temperature QAM and QAM-E show diverging adjoint loss (above 1e20) and task collapse, while TRQAM remains stable.
• TRQAM outperforms all baselines in offline-to-online RL and tightly tracks the target KL bound, while QAM with an external KL penalty drifts above the bound.

Surprising findings

• Tight KL budgets are best on most tasks, with puzzle-4x4 as the exception where larger budgets help.
• Gradient clipping does not prevent the adjoint-loss collapse of fixed-temperature QAM on Robomimic.

Limitations

• Relies on the terminal KL being approximated by the path-space KL estimator and on the assumption that AM reaches the tilted optimum.
• Best epsilon_KL varies by domain and requires per-domain tuning on tuning tasks.
• Antmaze-giant uses a time-varying KL schedule chosen by hand.

Experimental setup

Datasets: OGBench Robomimic

Benchmarks: OGBench Robomimic

Metrics: success rate

Baselines: FQL CGQL-Linex DSRL IFQL QAM QAM-E

Builds on: Q-learning with Adjoint Matching Adjoint Matching flow Q-learning DSRL Girsanov's theorem

Closest papers by meaning

• Aligning Flow Map Policies with Optimal $Q$-Guidance (NeurIPS 2026 · similarity 0.89)
• Flow Matching for Offline Reinforcement Learning with Discrete Actions (NeurIPS 2026 · similarity 0.89)
• Flow Q-Learning (ICML 2025 · similarity 0.88)
• Flow Matching Policy Optimization with Mirror Descent and Entropy Constraints (NeurIPS 2026 · similarity 0.87)
• Diffusion Policies Creating a Trust Region for Offline Reinforcement Learning (NeurIPS 2024 · similarity 0.87)
• Adversarial Policy Optimization for Offline Preference-based Reinforcement Learning (ICLR 2025 · similarity 0.87)