Research map / Reinforcement learning / Policy gradient and actor-critic

Long-Horizon Q-Learning: Accurate Value Learning via n-Step Inequalities

NeurIPS 2026 · Reinforcement learning › Policy gradient and actor-critic · approach: Temporal-difference and discounting theory · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes long-horizon Q-learning (LQL), which adds two-sided hinge penalties derived from n-step optimality inequalities to TD learning, reusing existing network outputs, and reports gains over 1-step TD and n-step TD on OGBench and RoboMimic.

Problem

Bootstrapped off-policy Q-learning is brittle over long horizons, as estimation errors compound through TD updates, while n-step targets become biased by logged off-policy actions.

Approach

Turns the optimality-tightening inequality (any realized action sequence lower-bounds the optimal value) into asymmetric hinge-squared penalties over replayed trajectory segments, computed from the same Q-network outputs used by the TD loss, so no auxiliary networks or extra forward passes are needed.

Key results

• On humanoidmaze-giant (L=64), LQL reaches 75.7% average success versus 0% for 1-step TD and 38.4% for best n-step TD (n=4), with n=64 TD at 6.1%.
• Averaged over OGBench groups, LQL with Best-of-N reaches 71.2% total success vs 48.6% for TD and 54.5% for TD-n.
• On RoboMimic with Best-of-N, LQL achieves 64.8% total vs 52.0% for TD; with FQL it reaches 75.5% vs 1.5% for TD.

Surprising findings

• 1-step TD blew up in 14 of 15 humanoidmaze-giant training runs, with average Q magnitude exceeding 900 despite non-positive rewards.
• Trajectory sampling alone sometimes outperformed standard transition-level TD, though removing the hinge penalties substantially hurt LQL.

Limitations

• Optimality inequality holds only in expectation, so per-sample penalties introduce bias under stochastic dynamics.
• Applied only to 1-step TD and action chunking; the n-step TD variant is left to future work.
• Evaluated on a limited set of OGBench and RoboMimic tasks with a fixed hinge coefficient default.

Experimental setup

Datasets: OGBench scene OGBench cube-double OGBench cube-triple OGBench humanoidmaze-md OGBench humanoidmaze-giant OGBench antmaze-giant RoboMimic can RoboMimic square

Benchmarks: OGBench RoboMimic

Metrics: success rate (%) Q-value magnitude stability

Baselines: 1-step TD n-step TD optimality tightening IQL ReBRAC

Builds on: optimality tightening TD learning flow Q-learning RLPD n-step TD

Closest papers by meaning

• Horizon Adaptive Offline Policy Learning via Value Stitching (NeurIPS 2026 · similarity 0.90)
• A Perspective of Q-value Estimation on Offline-to-Online Reinforcement Learning (AAAI 2024 · similarity 0.90)
• Target Networks and Over-parameterization Stabilize Off-policy Bootstrapping with Function Approximation (ICML 2024 · similarity 0.89)
• From Weak Data to Strong Policy: Q-Targets Enable Provable In-Context Reinforcement Learning (NeurIPS 2026 · similarity 0.89)
• Q-Distribution guided Q-learning for offline reinforcement learning: Uncertainty penalized Q-value via consistency model (NeurIPS 2024 · similarity 0.89)
• Q-SFT: Q-Learning for Language Models via Supervised Fine-Tuning (ICLR 2025 · similarity 0.88)