Research map / Reinforcement learning / Visual and representation RL
Proper Laplacian Representation Learning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes ALLO, a max-min augmented-Lagrangian objective with stop-gradients that makes the Laplacian representation the unique stable equilibrium, recovering both eigenvectors and eigenvalues without GGDO's untunable hyperparameters.
Problem
Approximating the Laplacian representation of an MDP with neural networks: existing graph-drawing objectives (GDO/GGDO) converge to arbitrary eigenvector rotations, depend on environment-specific hyperparameters that cannot be tuned without the true solution, and cannot recover eigenvalues.
Approach
Augment the graph drawing objective with dual variables (augmented Lagrangian) and stop-gradient operators that make the constraints asymmetric across eigenvectors; the resulting max-min objective ALLO has the sorted true eigenvectors as its only stable equilibrium, and the optimal dual variables equal the eigenvalues (up to a factor).
Key results
- ALLO finds close approximations of the true Laplacian eigenvectors across 12 grid environments, where GGDO fails completely in GridMaze-7 and GridMaze-9 and partially in GridMaze-32 and GridRoom-64.
- Differences are statistically significant (p<0.01) in 9 of 12 environments.
- ALLO's dual variables give more accurate eigenvalue estimates than the GGDO-based Monte Carlo alternative for the second to last eigenvalues across environments.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Replace GGDO with ALLO | Laplacian representation learning · 12 grid environments, (x,y) inputs | average cosine similarity | Helped |
| Replace the GGDO-based Monte Carlo alternative with ALLO | Laplacian eigenvalue estimation · 12 grid environments | relative error | Helped |
| Add a monotonically increasing barrier coefficient (ablation) | ALLO without barrier increase · GridMaze-19 | average cosine similarity | Helped |
| Increase the barrier coefficient at different rates (ablation) | ALLO · GridMaze-19 | average cosine similarity | Mixed |
| Add stop-gradient operators (ablation) | GGDO-style objective · GridMaze-19 | average cosine similarity | Helped |
| Add dual variables (ablation) | ALLO without dual variables · GridMaze-19 | average cosine similarity | No clear effect |
Surprising findings
- Standard augmented Lagrangian was previously dismissed for lack of empirical stability; the stop-gradient is what makes it work.
- Adding dual variables does not hurt eigenvector accuracy, though the authors conjecture they add stability in larger environments.
Limitations
- Theory is for the stationary/stable-equilibrium analysis; the effect of the barrier coefficient on optimization is not theoretically understood.
- Experiments are limited to grid-world environments with a uniform random policy and d=11.
- Dual variables' stabilizing benefit is conjectured but not shown empirically in the experiments.
Experimental setup
Datasets: grid environments generated from uniform random policy (GridRoom, GridMaze, GridRoomSym)
Benchmarks: GridRoom-4 / GridRoomSym-4 / GridMaze-7/9/19/32 / GridRoom-64 grid environments
Metrics: average cosine similarity to true eigenvectors relative error of eigenvalues p-value significance
Baselines: GDO GGDO GGDO with Monte Carlo eigenvalue estimation
Builds on: graph drawing objective generalized graph drawing objective augmented Lagrangian method spectral inference networks
Closest papers by meaning
- Online Laplacian-Based Representation Learning in Reinforcement Learning (ICML 2025 · similarity 0.94)
- Towards Better Laplacian Representation in Reinforcement Learning with Generalized Graph Drawing (ICML 2021 · similarity 0.94)
- The Laplacian in RL: Learning Representations with Efficient Approximations (ICLR 2019 · similarity 0.94)
- The Laplacian Keyboard: Beyond the Linear Span (NeurIPS 2026 · similarity 0.89)
- Diffusion Spectral Representation for Reinforcement Learning (NeurIPS 2024 · similarity 0.85)
- BeigeMaps: Behavioral Eigenmaps for Reinforcement Learning from Images (ICML 2024 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.