Research map / Reinforcement learning / Visual and representation RL
Abstract and Explore: A Novel Behavioral Metric with Cyclic Dynamics in Reinforcement Learning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes BCD, a behavioral metric combining successor features and vector-quantized latent transition dynamics with cyclic (forward, backward, inverse) dynamics losses to improve intrinsic-motivation exploration in sparse-reward procedurally-generated MiniGrid environments.
Problem
Intrinsic-motivation exploration in procedurally-generated, stochastic environments with uninformative extrinsic rewards suffers from state representation collapse and insufficient modeling of environment dynamics.
Approach
A bisimulation-style behavioral metric that measures state distance via both discrete vector-quantized one-step transition latents (immediate effect) and successor-feature cosine distance (cumulative effect), trained jointly with cyclic forward, backward and inverse dynamics losses.
Key results
- BCD reaches about 0.78 ± 0.182 return on ObstructedMaze-2Q and 0.94 ± 0.012 on KeyCorridor-S6R3 in ablation table, outperforming PPO-based baselines per Figure 3.
- Removing forward dynamics learning increases variance on ObstructedMaze-2Q (0.77 ± 0.245); removing backward dynamics slightly lowers performance (0.74 ± 0.288).
- Codebook size {4,8,16} and code length {16,32,64} have only slight impact on final results.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| remove forward dynamics learning (ablation) | BCD · ObstructedMaze-2Q | return: 0.78 ± 0.182 -> 0.77 ± 0.245 | No clear effect |
| remove forward dynamics learning (ablation) | BCD · KeyCorridor-S6R3 | return: 0.94 ± 0.012 -> 0.94 ± 0.039 | No clear effect |
| remove backward dynamics learning (ablation) | BCD · ObstructedMaze-2Q | return: 0.78 ± 0.182 -> 0.74 ± 0.288 | Hurt |
| remove backward dynamics learning (ablation) | BCD · KeyCorridor-S6R3 | return: 0.94 ± 0.012 -> 0.93 ± 0.021 | Hurt |
| vary codebook size (ablation) | BCD · MiniGrid environments | return | No clear effect |
| vary code length (ablation) | BCD · MiniGrid environments | return | No clear effect |
Limitations
- Evaluated only on MiniGrid procedurally-generated environments with small discrete observations.
- Ablation gains are small with overlapping variance across seeds.
Experimental setup
Datasets: MiniGrid procedurally-generated environments
Benchmarks: MiniGrid ObstructedMaze-2Q MiniGrid KeyCorridor-S6R3 MiniGrid ObstructedMaze-2Dlh MiniGrid ObstructedMaze-2Dlhb
Metrics: episodic return variance across seeds
Baselines: PPO ICM Bebold RIDE RND
Builds on: PPO Bebold RIDE RND ICM bisimulation metric
Closest papers by meaning
- Reward-Aware Proto-Representations in Reinforcement Learning (NeurIPS 2025 · similarity 0.88)
- Episodic Novelty Through Temporal Distance (ICLR 2025 · similarity 0.88)
- Reward-Consistent Dynamics Models are Strongly Generalizable for Offline Reinforcement Learning (ICLR 2024 · similarity 0.87)
- Rethinking Exploration in Reinforcement Learning with Effective Metric-Based Exploration Bonus (NeurIPS 2024 · similarity 0.86)
- Reinforcement Learning Under Latent Dynamics: Toward Statistical and Algorithmic Modularity (NeurIPS 2024 · similarity 0.86)
- Potential-Based Reward Shaping for Intrinsic Motivation (Student Abstract) (AAAI 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.