Research map / Reinforcement learning / Offline RL

From Weak Data to Strong Policy: Q-Targets Enable Provable In-Context Reinforcement Learning

NeurIPS 2026 · Reinforcement learning › Offline RL · approach: Decision Transformer sequence models · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes Q-Target Pretrained Transformers (QTPT), which pretrain a context-conditioned Transformer with Bellman-style Q-targets instead of supervised behavior cloning, with suboptimality bounds and controlled/D4RL/math experiments.

Problem

In-context RL pretraining with supervised behavior prediction ties the learned policy to offline action quality, so weak or suboptimal trajectories yield biased imitation targets.

Approach

Keep the context-conditioned Transformer for task inference but replace behavior-action cloning with a Bellman/TD Q-target regression on rewards and transitions in the context, so the policy is greedy over learned action values rather than imitating the behavior policy.

Key results

• On the stochastic linear bandit, QTPT consistently outperforms SPT under both random and LinUCB pretraining data (averaged over 1000 runs).
• On D4RL AntMaze-umaze-v2 under a matched Transformer protocol, QTPT reports 35.94 ± 13.52% active-stage success vs 26.96 ± 11.39% for SPT (3–4 seeds, descriptive).
• On GSM8K, QTPO improves over behavior cloning across all mixture ratios by an average of 1.81%.

Surprising findings

• Random-data pretraining outperforms expert-data pretraining in Darkroom and Miniworld, attributed to broader state-action support.
• QTPT-NoCtx (no context conditioning) drops to near zero reward (0.38 on random data), and adding a CQL penalty gives no statistically reliable improvement in the overestimation diagnostic.

Limitations

• Theory requires coverage/support conditions on offline data and does not cover arbitrary low-coverage datasets.
• D4RL results are fixed-protocol descriptive extensions with high between-seed variance, not SOTA comparisons.
• Bound contains a worst-case T^2 dependence and approximation floor epsilon_tf that does not vanish with n for a fixed misspecified Transformer.

Experimental setup

Datasets: offline random-policy bandit trajectories offline LinUCB bandit trajectories Darkroom offline data Dark Key-to-door offline data Miniworld offline data D4RL Kitchen-partial-v0 D4RL Kitchen-mixed-v0 D4RL AntMaze-umaze-v2 GSM8K

Benchmarks: stochastic linear bandit Darkroom Dark Key-to-door Miniworld D4RL Kitchen D4RL AntMaze GSM8K

Metrics: suboptimality gap average cumulative reward normalized score success rate accuracy improvement over behavior cloning

Baselines: SPT Q-SFT QDT QTPT-MC QTPT-NoCtx RL^2 PEARL CQL IQL behavior cloning

Builds on: Decision Pretrained Transformer Algorithm Distillation Q-SFT Q-learning Decision Transformer Conservative Q-Learning

Closest papers by meaning

• In-Context Reinforcement Learning From Suboptimal Historical Data (ICML 2025 · similarity 0.91)
• Q-value Regularized Transformer for Offline Reinforcement Learning (ICML 2024 · similarity 0.90)
• Transformers as Decision Makers: Provable In-Context Reinforcement Learning via Supervised Pretraining (ICLR 2024 · similarity 0.90)
• Transformers Provably Implement In-Context Reinforcement Learning with Policy Improvement (NeurIPS 2026 · similarity 0.89)
• Long-Horizon Q-Learning: Accurate Value Learning via n-Step Inequalities (NeurIPS 2026 · similarity 0.89)
• Latent Q-Barrier Shielding for Safe In-Context Reinforcement Learning (NeurIPS 2026 · similarity 0.88)