Research map / Reinforcement learning / Visual and representation RL

BeigeMaps: Behavioral Eigenmaps for Reinforcement Learning from Images

ICML 2024 · Reinforcement learning › Visual and representation RL · approach: Learned state and action representations

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes BeigeMaps, locality-preserving behavioral eigenmap representations (NeuralEF eigenfunctions of behavioral-distance kernels) that drop into existing behavioral-distance RL algorithms to improve pixel-based control on DMC.

Problem

Learning representations from high-dimensional image observations for RL via behavioral (bisimulation-style) distances; existing isometric embedding objectives may be ill-defined and unstable when the behavioral distance is not realizable in low dimensions.

Approach

Replace the isometric (Isomap) embedding objective with a locality-preserving Laplacian eigenmap objective computed via NeuralEF on Gaussian kernels of behavioral distances, allowing long-range distortion while preserving local structure and inducing cluster-like representations.

Key results

• BeigeMap variants considerably outperform their baselines for DBC, R-DBC, and KSME in IQM normalized return across 7 DMC environments.
• RAP+BeigeMap obtains higher average returns than RAP, though with confidence-interval overlap.
• The worst BeigeMap variant (R-DBC+BeigeMap) still outperforms an average baseline model.

Measured effects reported in the paper

Surprising findings

• DBC+BeigeMaps outperforms R-DBC, which was designed with explicit stability fixes, suggesting the orthonormality constraints mitigate embedding explosion and collapse.

Limitations

• Robustness to task-irrelevant distractors not evaluated; suggested as future work
• Replay buffer reduced from 1,000,000 to 100,000 due to computational constraints
• Gains for RAP+BeigeMap have overlapping confidence intervals

Experimental setup

Datasets: DeepMind Control Suite

Benchmarks: DeepMind Control Suite (finger spin, walker stand, walker walk, cheetah run, cartpole swingup, cartpole balance, ball in

Metrics: interquartile mean normalized return performance profiles 95% stratified bootstrap CI

Baselines: DBC R-DBC MICO KSME RAP SAC

Builds on: DBC (Deep Bisimulation for Control) NeuralEF Laplacian eigenmaps KSME SAC

Closest papers by meaning

• PAMD: Structured Adaptive Distances for Bisimulation Representations in Visual Reinforcement Learning (ICML 2026 · similarity 0.89)
• Learning Invariant Representations for Reinforcement Learning without Reconstruction (ICLR 2021 · similarity 0.89)
• Online Laplacian-Based Representation Learning in Reinforcement Learning (ICML 2025 · similarity 0.88)
• METRA: Scalable Unsupervised RL with Metric-Aware Abstraction (ICLR 2024 · similarity 0.86)
• SiT: Symmetry-invariant Transformers for Generalisation in Reinforcement Learning (ICML 2024 · similarity 0.86)
• Towards Principled Representation Learning from Videos for Reinforcement Learning (ICLR 2024 · similarity 0.86)