Research map / Reinforcement learning / Visual and representation RL
Maximum Total Correlation Reinforcement Learning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes maximum total correlation RL (MTC-RL), which adds a trajectory total-correlation bonus to SAC via a variational lower bound on latent state and action consistency, yielding more periodic, compressible, and robust policies.
Problem
Standard RL policies can be brittle to observation noise, action noise, and dynamics changes; the paper seeks a trajectory-level inductive bias that promotes simple, consistent, compressible behavior.
Approach
Augment the RL objective with the total correlation of latent states and actions within trajectories, lower-bounded by history-based latent dynamics and action prediction models, and implement this as a reward regularizer on SAC with the coefficient alpha tuned as a Lagrange multiplier against a target bound I_p.
Key results
- On eight DMC tasks at 1M steps, MTC scores 933 on Hopper Stand vs 683 for SAC and 985 on Finger Spin vs 955 for SAC.
- MTC trajectories compress more efficiently (smaller bzip2 files) than baselines, and its t-step action prediction NLL on Finger Spin is lowest at all horizons (e.g., 0.22 vs 0.78 for SAC at t=3).
- MTC achieves the best aggregated return under Gaussian observation noise and action noise, and higher returns under small body-mass changes than baselines.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| maximize trajectory total correlation with MTC | SAC on DMC continuous-control tasks · DMC tasks (eight-task comparison) | score | Helped |
| maximize trajectory total correlation with MTC | RL policies · observation perturbations across eight DMC tasks | normalized mean reward | Helped |
| maximize trajectory total correlation with MTC | RL policies · action perturbations across DMC tasks | average reward | Helped |
| maximize trajectory total correlation with MTC | RL policies · small mass changes across eight DMC tasks | average score | Helped |
| maximize trajectory total correlation with MTC | Walker Stand policies · spurious state correlations | reward | Helped |
| remove action prediction model KL regularization (MTC-NoA) (ablation) | MTC · Cheetah Run; action noise and mass changes | robustness | Hurt |
| remove state dynamics model KL regularization (SAC) (ablation) | MTC · Walker Stand and Cheetah Run; perturbations | robustness | Hurt |
| increase the total-correlation constraint I_p (ablation) | MTC · Walker Stand | unperturbed reward | No clear effect |
| increase the total-correlation constraint I_p (ablation) | MTC · Walker Stand | trajectory encoding size | Helped |
| increase the total-correlation constraint I_p (ablation) | MTC · Walker Stand under observation, action, and dynamics perturbations | robustness | Helped |
| use history-based policy observations (ablation) | MTC · Finger Spin | score: 985 $\pm$ 2 -> 986 $\pm$ 3 | No clear effect |
| use history-based policy observations (ablation) | SAC · Finger Spin | score: 955 $\pm$ 18 -> 951 $\pm$23 | No clear effect |
Surprising findings
- The lower bound on total correlation is always negative, so it cannot estimate total correlation itself but is still effective for optimization.
- Providing history-based observations to SAC or MTC did not change performance on Finger Spin, indicating gains come from the objective rather than non-Markovian inputs.
Limitations
- The variational lower bound is vacuous (always negative) and so cannot estimate the true total correlation.
- Hyperparameter I_p and history length must be tuned per task family.
- Mixed results on some tasks: on Acrobot Swingup and Walker Stand MTC does not clearly beat SAC, and on Metaworld it is only comparable.
Experimental setup
Datasets: DeepMind Control Suite Metaworld Planet image-based DMC tasks
Benchmarks: DeepMind Control Suite Metaworld Planet image-based DMC
Metrics: episode return success rate normalized return under perturbation bzip2 compressed file size negative log-likelihood of t-step-ahead action prediction
Baselines: SAC RPC RPC-Orig LZ-SAC SPAC CURL SAC-AE
Builds on: soft actor-critic RPC information bottleneck
Closest papers by meaning
- Oracle-Efficient Reinforcement Learning for Max Value Ensembles (NeurIPS 2024 · similarity 0.87)
- Reinforcement Learning with Simple Sequence Priors (NeurIPS 2023 · similarity 0.87)
- Foundation Policies with Hilbert Representations (ICML 2024 · similarity 0.86)
- Learning Self-Imitating Diverse Policies (ICLR 2019 · similarity 0.86)
- Iteratively Refined Behavior Regularization for Offline Reinforcement Learning (NeurIPS 2024 · similarity 0.86)
- Behavior-Regularized Diffusion Policy Optimization for Offline Reinforcement Learning (ICML 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.