Research map / Reinforcement learning / Visual and representation RL
State Chrono Representation for Enhancing Generalization in Reinforcement Learning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes State Chrono Representation (SCR), a metric-based RL representation learning method that adds chronological embeddings and temporal reward measurements over future states to bisimulation/MICo-style metric learning, evaluated on distracting DM_Control and Meta-World.
Problem
Metric-based (bisimulation/MICo) state representations for image-based RL struggle with demanding generalization under visual distractions and sparse or non-informative rewards because they rely on one-step transitions and miss long-term information.
Approach
Augments MICo-style state metric learning with a chronological embedding psi(x_i, x_j) trained under a chronological behavioral metric, plus a temporal measurement m(x_i, x_j) of cumulative rewards bounded by lower and upper constraints, and a diffuse-metric distance sqrt(|a|^2+|b|^2-a^T b) that avoids arccos and division.
Key results
- On the distraction setting of DM_Control at 500K steps, SCR achieves the best score on all eight tasks, e.g., 565.1 on cartpole-swing_up vs 218.4 for SAC and 185.7 on cartpole-swing_up_sparse where other metric methods are near zero.
- On six Meta-World tasks, SCR reaches an average success rate of 0.969 versus 0.886 for DrQ and 0.495 for SAC.
- In the default DM_Control setting, SCR is comparable to DrQ and SimSR rather than clearly superior.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add State Chrono Representation to SAC | SAC · DM_Control distraction, all eight tasks | score | Helped |
| add State Chrono Representation to SAC | SAC · DM_Control default, eight tasks | score | Mixed |
| add State Chrono Representation to SAC | SAC · Meta-World, six tasks | success rate | Helped |
| remove chronological embedding ψ (ablation) | SCR · DM_Control distraction, cheetah-run and walker-walk | score | Hurt |
| remove temporal measurement approximation m̂ (ablation) | SCR · DM_Control distraction, cheetah-run and walker-walk | score | Hurt |
| remove chronological embedding loss and temporal measurement loss, keeping only Lφ (ablation) | SCR · DM_Control distraction, cheetah-run and walker-walk | score | Hurt |
| replace the proposed distance d̂ with cosine distance (ablation) | SCR · DM_Control distraction, cheetah-run and walker-walk | score | Hurt |
| replace the proposed distance d̂ with MICo angular distance (ablation) | SCR · DM_Control distraction, cheetah-run and walker-walk | score | Hurt |
| replace the proposed distance d̂ with L1 distance (ablation) | SCR · DM_Control distraction, cheetah-run and walker-walk | score | Hurt |
| sample temporal step counts uniformly from [1, 100] instead of alternative tested ranges or fixed counts (ablation) | SCR · DM_Control distraction, cheetah-run and walker-walk | score | Helped |
Surprising findings
- Plain DrQ outperforms the metric-based baselines (DBC, MICo, SimSR) under camera-pose distraction, which the authors attribute to random cropping aligning the robot.
- Combining SCR with DrQ (SCR w/ DrQ) further raises distraction-setting scores substantially, e.g., 936.0 on ball_in_cup-catch.
Limitations
- Does not address truly partially observable MDPs.
- Evaluation limited to simulated DM_Control and Meta-World tasks with 84x84 pixel observations.
- Reported std is computed across per-run means over only 5 to 10 seeds.
Experimental setup
Datasets: DeepMind Control Suite Distracting Control Suite DAVIS2017 Meta-World
Benchmarks: DeepMind Control Suite (default) Distracting Control Suite Meta-World
Metrics: episode return success rate aggregate normalized score
Baselines: SAC DrQ DBC MICo SimSR RAP CBM
Builds on: DBC MICo SimSR SAC Interval Quasimetric Embedding
Closest papers by meaning
- Learning Latent Dynamic Robust Representations for World Models (ICML 2024 · similarity 0.87)
- Bridging State and History Representations: Understanding Self-Predictive RL (ICLR 2024 · similarity 0.86)
- A Generalized Bisimulation Metric of State Similarity between Markov Decision Processes: From Theoretical Propositions to Applications (NeurIPS 2025 · similarity 0.86)
- Sliding Puzzles Gym: A Scalable Benchmark for State Representation in Visual Reinforcement Learning (ICML 2025 · similarity 0.86)
- LLM-Empowered State Representation for Reinforcement Learning (ICML 2024 · similarity 0.85)
- Towards Principled Representation Learning from Videos for Reinforcement Learning (ICLR 2024 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.