Research map / Architectures and efficiency / Attention and transformers
TQL: Scaling Q-Functions with Transformers by Preventing Attention Collapse
Perry Dong, Kuo-Han Hung, Alexander Swerdlow, Dorsa Sadigh, Chelsea Finn
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add attention entropy loss with target entropy (ablation) | Transformer Q-function (26M) · cube-double (5 tasks) | success rate | Improved |
| guide entropy toward a target instead of fixed max-entropy penalty (ablation) | TQL (26M) · cube-double (5 tasks) | success rate | Improved |
| use per-layer and per-token temperatures instead of a single temperature (ablation) | TQL (26M) · cube-double (5 tasks) | success rate | Improved |
| scale critic from 0.4M to 26M parameters | prior offline RL methods (MLP, flow, transformer) · OGBench | success rate: average decrease of 10.6% | Hurt |
| scale plain transformer critic from 7M to 26M | Transformer baseline without TQL · OGBench | success rate | Hurt |
| TQL vs floq on OGBench overall average | TQL · OGBench 25 tasks | success rate: 34 -> 40 | Improved |
| TQL vs puzzle-3x3 floq | TQL · puzzle-3x3 | success rate: 35 -> 32 | Hurt |
Closest papers by meaning
- Stop Regressing: Training Value Functions via Classification for Scalable Deep RL (ICML 2024 · similarity 0.88)
- Compute-Optimal Scaling for Value-Based Deep RL (NeurIPS 2025 · similarity 0.87)
- ShiQ: Bringing back Bellman to LLMs (NeurIPS 2025 · similarity 0.86)
- Mixtures of Experts Unlock Parameter Scaling for Deep RL (ICML 2024 · similarity 0.86)
- Penalizing Infeasible Actions and Reward Scaling in Reinforcement Learning with Offline Data (ICML 2025 · similarity 0.86)
- Q-SFT: Q-Learning for Language Models via Supervised Fine-Tuning (ICLR 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
