Research map / Reinforcement learning / MDPs and planning
The Indoor-Training Effect: Unexpected Gains from Distribution Shifts in the Transition Function
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Shows that tabular Q-learning and SARSA agents trained on a noise-free MDP can outperform agents trained on the same noisy target MDP, using a Noise Injection method that perturbs transition functions of ATARI-style grid games (PacMan, Pong, Breakout).
Problem
Conventional wisdom holds that training and testing on the same MDP yields the best performance; the paper asks whether distribution shifts in transition probabilities between training and testing environments can instead improve zero-shot policy transfer.
Approach
Generate controlled variants of an MDP by adding parametric Gaussian noise to the transition function (Noise Injection, producing delta-environments whose noise level acts as a distance metric), and compare a Learnability agent trained and tested on the noisy MDP against a Generalization agent trained on the noise-free MDP; also adds semantic variants (teleporting ghosts, following paddles). The g
Key results
- Generalization agents trained on the noise-free MDP outperformed Learnability agents on the noisy target in most reported settings (two-sided t-test, p<0.001), across 60 MDP variations.
- Spearman correlation between exploration divergence D_LG and reward gap R_LG was 0.43 for PacMan and 0.26 for Pong (p<0.005).
- The effect also appeared under semantic variations, e.g., RandomGhost-trained agents beating TeleportingGhost-trained agents on TeleportingGhost PacMan, though with smaller margins in Pong.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Train on the noise-free target MDP instead of the same noisy δ-environment | SARSA agents on PacMan v4 · PacMan v4, low- and high-noise δ-environments | mean reward | Helped |
| Train on the noise-free target MDP instead of the same noisy δ-environment | Q-Learning agents on PacMan v4 · PacMan v4, low- and high-noise δ-environments | mean reward | Helped |
| Train on the noise-free target MDP instead of the same noisy δ-environment | SARSA and Q-Learning agents · PacMan v2 and v3 | mean reward | Helped |
| Train on the noise-free target MDP instead of the same noisy δ-environment | SARSA and Q-Learning agents · Pong p1 and p2 | mean reward | Mixed |
| Train on the noise-free target MDP instead of the same noisy δ-environment | SARSA and Q-Learning agents · Breakout b1 and b2 | mean reward | Helped |
| Train with RandomGhost instead of TeleportingGhost, then test with TeleportingGhost | PacMan agents · PacMan v2 and v3, TeleportingGhost | mean reward | Helped |
| Train with DirectionalPaddle instead of RandomPaddle, then test with RandomPaddle | Pong agents · Pong p1 and p2, RandomPaddle | mean reward | Helped |
| Use high noise rather than low noise | Learnability and Generalization agents · PacMan v4, noisy δ-environments | mean reward | Hurt |
| Have substantial divergence in explored state-action pairs between Learnability and Generalization agents | Generalization Agent relative to Learnability Agent · PacMan and Pong MDPs | mean reward | Hurt |
| Increase exploration-pattern divergence between agents | Generalization Agent relative to Learnability Agent · PacMan | Spearman correlation with reward gap: 0.43 (p<0.005) | Helped |
| Increase exploration-pattern divergence between agents | Generalization Agent relative to Learnability Agent · Pong | Spearman correlation with reward gap: 0.26 (p<0.005) | Helped |
Surprising findings
- Agents trained on the noise-free environment can outperform agents trained on the same noisy target environment they are tested on.
- The Generalization advantage persisted under semantic changes to ghost and paddle behaviour, not only Gaussian noise.
Limitations
- Experiments limited to ATARI-style games and tabular/classical RL; deep RL only as a supplementary DQN check.
- No real-world or robotics validation; robotics implications are speculative.
- Noise injection is applied to transition probabilities that must be enumerated explicitly, which restricts scale.
Experimental setup
Datasets: PacMan MDPs (v2, v3, v4 grids) Pong MDPs (p1, p2 grids) Breakout MDPs (b1, b2, b3 grids)
Benchmarks: PacMan Pong Breakout (custom ATARI-style grid variants)
Metrics: mean reward over episodes area under curve with regret normalization two-sided t-test Spearman correlation between exploration divergence and rewa
Baselines: Learnability agent (trained and tested on the same delta-environment)
Builds on: Q-learning SARSA DQN Berkeley PacMan Projects
Closest papers by meaning
- Offline-Online Reinforcement Learning for Linear Mixture MDPs (NeurIPS 2026 · similarity 0.85)
- Emergence of In-Context Reinforcement Learning from Noise Distillation (ICML 2024 · similarity 0.85)
- Towards Principled Representation Learning from Videos for Reinforcement Learning (ICLR 2024 · similarity 0.85)
- Learning When to Stop: Selective Imitation Learning Under Arbitrary Dynamics Shift (NeurIPS 2026 · similarity 0.84)
- Abstract and Explore: A Novel Behavioral Metric with Cyclic Dynamics in Reinforcement Learning (AAAI 2024 · similarity 0.84)
- Forecasting in Offline Reinforcement Learning for Non-stationary Environments (NeurIPS 2025 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.