Research map / Reinforcement learning / Visual and representation RL
Horizon Generalization in Reinforcement Learning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Formalizes horizon generalization and planning invariance in goal-conditioned RL, proves quasimetric policies exhibit both, and runs tabular and high-dimensional experiments (ant, AntMaze, humanoid) comparing CRL, SAC, and quasimetric architectures.
Problem
Goal-conditioned RL policies trained only on nearby goals often fail on distant goals; the paper asks whether policies can generalize over the horizon from short to long start-goal pairs without changing dynamics or observations.
Approach
Shows that horizon generalization is tied to invariance to planning: a quasimetric policy that is optimal on nearby pairs is, by induction over doubling radii and the triangle inequality, optimal everywhere, and measures this via a horizon generalization parameter eta and planning-invariance ratios.
Key results
- In the Ant environment, CMD-1 architecture achieves eta approximately 0.8, well above the critical value 0.5 needed for unbounded reach, while L2-distance architectures reach about 0.5.
- A metric regression with quasimetric (path relaxation) policy achieves near-perfect horizon generalization in the tabular maze.
- Horizon generalization parameter eta correlates well with planning-invariance measures across CRL variants and architectures; the random policy is planning-invariant but fails to generalize.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use CMD-1 with its quasimetric architecture instead of CRL with a standard architecture | Goal-conditioned RL · Function approximation horizon generalization | horizon generalization parameter η: η ≈ 0.8 | Helped |
| Use an L2-distance architecture instead of an MLP architecture | Contrastive RL · Function approximation horizon generalization | horizon generalization parameter η: η≈ 0.5 | Helped |
| Use a quasimetric architecture (CMD-1) instead of a standard architecture (CRL) | Goal-conditioned RL · AntMaze and Humanoid | horizon generalization parameter η | Helped |
| Use the L2-distance architecture instead of the MLP architecture | Contrastive RL · Bellman error experiment | Bellman loss | Helped |
| Apply path relaxation to the empirical hitting-time distances | Metric regression policy · Tabular navigation task | horizon generalization | Helped |
| Use a random policy instead of a goal-directed policy | Goal-conditioned navigation · Function approximation horizon generalization | horizon generalization | Hurt |
Surprising findings
- A random policy is perfectly planning invariant yet shows poor horizon generalization, because the base case of succeeding on nearby goals fails.
- Lower Bellman error is observed for the architecture with stronger horizon generalization, even though neither method is trained with a Bellman loss.
Limitations
- Horizon generalization is assumed to hold everywhere as a binary base case; practical algorithms achieve it only on a spectrum.
- Experiments are on simulated navigation and locomotion tasks; no results for robotic manipulation or language agents.
- No algorithm achieves perfect horizon generalization in high-dimensional settings.
Experimental setup
Datasets: simulated didactic maze enlarged Ant quadruped environment AntMaze Humanoid
Benchmarks: didactic tabular maze enlarged Ant navigation AntMaze Humanoid
Metrics: success rate stratified by distance horizon generalization parameter eta planning invariance ratio (success with vs without planning) Bellman error
Baselines: CRL SAC CMD-1 metric regression random policy
Builds on: contrastive RL metric residual network contrastive metric distillation successor distance
Closest papers by meaning
- DSAP: Enhancing Generalization in Goal-Conditioned Reinforcement Learning (AAAI 2026 · similarity 0.90)
- On Generalization Across Environments In Multi-Objective Reinforcement Learning (ICLR 2025 · similarity 0.89)
- Flattening Hierarchies with Policy Bootstrapping (NeurIPS 2025 · similarity 0.88)
- Closing the Gap between TD Learning and Supervised Learning - A Generalisation Point of View. (ICLR 2024 · similarity 0.87)
- Diffusion Subgoal Planning for Long-Horizon Offline Goal-Conditioned Reinforcement Learning (NeurIPS 2026 · similarity 0.87)
- Unifying Goal-Conditioned RL and Unsupervised Skill Learning via Control-Maximization (NeurIPS 2026 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.