Research map / Reinforcement learning / Offline RL
Towards General-Purpose Model-Free Reinforcement Learning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes MR.Q, a general model-free deep RL algorithm that learns model-based state-action representations approximately linear in the value function, evaluated with one hyperparameter set across Gym, DMC (proprioceptive and visual) and Atari.
Problem
General-purpose RL: most deep RL algorithms are tailored to specific benchmarks with tuned hyperparameters, while model-based general methods (DreamerV3, TD-MPC2) are computationally costly and complex.
Approach
Learn state-action embeddings with model-based objectives (reward, next-state embedding via target encoder, terminal, unrolled over a short horizon) so the value function is approximately linear in the features, then use a non-linear value head; combined with TD3-style learning, categorical reward loss, LAP and multi-step returns, under one fixed hyperparameter set.
Key results
- MR.Q achieves the highest performance on both DMC benchmarks (proprioceptive and visual) with a single hyperparameter set.
- MR.Q falls slightly behind TD7 on Gym locomotion but is the strongest overall across continuous control benchmarks.
- On Atari, MR.Q surpasses PPO, DQN and Rainbow, while DreamerV3 outperforms MR.Q using a model with about 40x more parameters.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use a nonlinear rather than linear value function (ablation) | MR.Q · Gym - Locomotion | TD3-normalized performance: -1.17 | Helped |
| use a nonlinear rather than linear value function (ablation) | MR.Q · Atari - 1M | Human-normalized performance: -1.35 | Helped |
| use a state embedding rather than a state-action embedding as dynamics target (ablation) | MR.Q · Gym - Locomotion | TD3-normalized performance: -0.10 | Helped |
| use a target encoder for the dynamics target (ablation) | MR.Q · Gym - Locomotion | TD3-normalized performance: -0.53 | Helped |
| avoid reverting simultaneously to linear value and current-encoder state-action targets (ablation) | MR.Q · Gym - Locomotion | TD3-normalized performance: -1.47 | Helped |
| use a linear MDP predictor rather than separate nonlinear prediction networks (ablation) | MR.Q · Gym - Locomotion | TD3-normalized performance: -0.01 | No clear effect |
| use categorical rather than MSE reward loss (ablation) | MR.Q · Atari - 1M | Human-normalized performance: -0.79 | Helped |
| use reward scaling (ablation) | MR.Q · Atari - 1M | Human-normalized performance: 0.18 | No clear effect |
| use the minimum rather than mean of target value functions (ablation) | MR.Q · Gym - Locomotion | TD3-normalized performance: -0.09 | Helped |
| use prioritized sampling and Huber value loss (ablation) | MR.Q · Gym - Locomotion | TD3-normalized performance: -0.10 | Helped |
| use model-based representation learning (ablation) | MR.Q · Gym - Locomotion | TD3-normalized performance: -0.56 | Helped |
| use multi-step rather than one-step value returns (ablation) | MR.Q · Gym - Locomotion | TD3-normalized performance: -0.33 | Helped |
Surprising findings
- Increasing model capacity with a non-linear model does not improve performance, suggesting maintaining an approximately linear value relationship matters more than capacity.
- Top performers on Gym and DMC fail to transfer to the other benchmark, and DreamerV3 fails to learn Dog and Humanoid DMC tasks.
Limitations
- Not equipped for hard exploration tasks or non-Markovian environments.
- Evaluation limited to standard RL benchmarks; no testing in settings like team video games, drone racing, or LLMs.
- Performance falls slightly behind TD7 on Gym locomotion and behind DreamerV3 on Atari.
Experimental setup
Datasets: Gym locomotion (MuJoCo v4) DeepMind Control Suite (28 tasks) Atari-57 (Arcade Learning Environment)
Benchmarks: Gym locomotion DMC proprioceptive DMC visual Atari-57
Metrics: TD3-normalized score average episode reward human-normalized score
Baselines: TD7 TD-MPC2 DreamerV3 PPO DrQ-v2 DQN Rainbow
Builds on: TD3 TD7 LAP DreamerV3 TD-MPC2
Closest papers by meaning
- A Clean Slate for Offline Reinforcement Learning (NeurIPS 2025 · similarity 0.91)
- The Edge-of-Reach Problem in Offline Model-Based Reinforcement Learning (NeurIPS 2024 · similarity 0.89)
- Model-based RL as a Minimalist Approach to Horizon-Free and Second-Order Bounds (ICLR 2025 · similarity 0.89)
- Model-Based Transfer Learning for Contextual Reinforcement Learning (NeurIPS 2024 · similarity 0.89)
- Tricks or Traps? A Deep Dive into RL for LLM Reasoning (ICLR 2026 · similarity 0.89)
- Dual RL: Unification and New Methods for Reinforcement and Imitation Learning (ICLR 2024 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.