Research map / Reinforcement learning / Visual and representation RL

Online Laplacian-Based Representation Learning in Reinforcement Learning

ICML 2025 · Reinforcement learning › Visual and representation RL · approach: Learned state and action representations

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes AGDO, an asymmetric graph drawing objective for learning Laplacian representations online alongside policy updates, with a convergence proof under bounded policy drift and grid-world experiments using PPO, VPG and DQN.

Problem

Laplacian-based representations in RL are usually learned under a fixed (uniform) policy, so they become stale as the policy changes; whether they can be learned online with theoretical guarantees alongside policy learning was an open question.

Approach

Uses an asymmetric graph drawing objective (AGDO), a simplification of ALLO that uses stop-gradient asymmetry and no dual variables, and analyzes online projected gradient descent on a sequence of time-varying AGDO losses, showing ergodic convergence to a critical point under bounded policy drift.

Key results

• In a fixed uniform policy setting, AGDO and ALLO achieve almost identical average cosine similarity to the true Laplacian representation.
• In the online setting with PPO, cosine similarity trends upward over training across all four grid environments for both AGDO and ALLO.
• Ablation shows smaller PPO clipping (more bounded drift) yields higher representation accuracy, while DQN's representation accuracy is much lower than on-policy methods.

Measured effects reported in the paper

Surprising findings

• Increasing the number of encoder update steps per sample did not improve accuracy, attributed to replay buffer sampling noise.
• DQN's epsilon-greedy policy changes drastically when the greedy action switches, yielding much lower representation accuracy than on-policy methods.

Limitations

• Convergence guarantee relies on bounded policy drift and ergodic/unique stationary distribution assumptions.
• Empirical evaluation limited to small grid-world environments.
• Drift bound increases with number of states, slowing convergence in larger environments.

Experimental setup

Datasets: grid world environments (GridRoom-1, GridRoom-4, GridMaze-9, GridMaze-11)

Benchmarks: GridRoom-1 GridRoom-4 GridMaze-9 GridMaze-11

Metrics: average cosine similarity to true Laplacian representation total reward

Baselines: ALLO DQN VPG PPO with different clipping

Builds on: ALLO (augmented Lagrangian Laplacian objective) Wu et al. 2018 Laplacian in RL generalized graph drawing objective PPO deep covering eigenoptions

Closest papers by meaning

• Towards Better Laplacian Representation in Reinforcement Learning with Generalized Graph Drawing (ICML 2021 · similarity 0.95)
• Proper Laplacian Representation Learning (ICLR 2024 · similarity 0.94)
• The Laplacian Keyboard: Beyond the Linear Span (NeurIPS 2026 · similarity 0.89)
• Diffusion Spectral Representation for Reinforcement Learning (NeurIPS 2024 · similarity 0.89)
• BeigeMaps: Behavioral Eigenmaps for Reinforcement Learning from Images (ICML 2024 · similarity 0.88)
• Towards Principled Representation Learning from Videos for Reinforcement Learning (ICLR 2024 · similarity 0.87)