Research map / Reinforcement learning / Offline RL
Scrutinize What We Ignore: Reining In Task Representation Shift Of Context-Based Offline Meta Reinforcement Learning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Links context-based offline meta-RL's encoder/policy alternating optimization to return-discrepancy bounds, identifies "task representation shift" as a missing monotonicity condition, and proposes controlling encoder update frequency/count to rein it in across three I(Z;M) objectives.
Problem
Context-based offline meta-RL methods alternate encoder and policy training to maximize I(Z;M), but the theory for why this improves performance is underexplored and ignores how the task representation changes during alternating updates.
Approach
Extends the return discrepancy scheme to COMRL to show alternating I(Z;M) encoder updates and offline RL policy updates lift a lower bound on expected return; adds a task representation shift term to the monotonicity condition and controls when/how often the encoder is updated (N_k, N_acc) to guarantee monotonic improvement.
Key results
- Across benchmarks, the best task-representation-shift setting shows statistically significant gains over the worst setting on 20 testing tasks, and the original alternating setting (N_k=1, N_acc=1) fails to reach the best asymptotic performance.
- On Ant-Dir with random, medium, and expert data, reining in task representation shift improves performance for contrastive, reconstruction, and cross-entropy objectives, except contrastive on random data.
- Settings with N_k>1 and N_acc=1 more often beat the original setting than settings with N_acc>1, so the authors recommend tuning N_k first, which also reduces training time.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Explicitly control context-encoder updates to rein in task representation shift | Context-based offline meta-RL · MuJoCo and MetaWorld test tasks | testing return | Helped |
| Use N_k > 1 and N_acc = 1 instead of the original N_k = 1 and N_acc = 1 (ablation) | Task representation shift control · MuJoCo and MetaWorld benchmarks | testing return | Helped |
| Use N_k = 1 and N_acc > 1 instead of the original N_k = 1 and N_acc = 1 (ablation) | Task representation shift control · MuJoCo and MetaWorld benchmarks | testing return | Mixed |
| Use the original update schedule, N_k = 1 and N_acc = 1, rather than a better-performing shift-control setting (ablation) | Task representation shift control · MuJoCo and MetaWorld benchmarks | asymptotic testing return | Hurt |
| Rein in task representation shift | Contrastive-based, reconstruction-based, and cross-entropy-based objectives · Ant-Dir, medium and expert datasets | testing return | Helped |
| Apply the contrastive-based objective on random-quality data | Contrastive-based context encoder · Ant-Dir, random dataset | testing return | Hurt |
| Pre-train the context encoder rather than train it from scratch (ablation) | Cross-entropy-, contrastive-, and reconstruction-based context encoders · Ant-Dir | testing return | Mixed |
Surprising findings
- Pre-training the context encoder and then training the policy without updating the encoder showed a large performance gap versus training from scratch, with the theory suggesting pre-training cannot guarantee monotonic improvement.
- Less desirable t-SNE task visualizations can still correspond to better performance, so task-representation visualization does not reliably imply asymptotic performance.
Limitations
- Relies on Lipschitz policy assumption and assumptions on discrete, limited task representation space and bounded encoder-update discrepancy that are only partially justified empirically.
- Threshold beta and update schedule are set heuristically; a smarter automatic control method is left to future work.
- Focuses only on the representation part; policy learning conditioned on the optimal representation is not addressed.
Experimental setup
Datasets: MuJoCo Ant-Dir MuJoCo Walker-Param MetaWorld Reach MetaWorld Button-Press MetaWorld Dial-Turn MetaWorld Push
Benchmarks: MuJoCo Ant-Dir MuJoCo Walker-Param MetaWorld Reach MetaWorld Button-Press MetaWorld Dial-Turn MetaWorld Push
Metrics: testing return paired t-test significance
Baselines: FOCAL CORRO UNICORN CSRO
Builds on: FOCAL CORRO UNICORN BRAC return discrepancy
Closest papers by meaning
- Towards an Information Theoretic Framework of Context-Based Offline Meta-Reinforcement Learning (NeurIPS 2024 · similarity 0.95)
- Entropy Regularized Task Representation Learning for Offline Meta-Reinforcement Learning (AAAI 2025 · similarity 0.93)
- Generalizable Task Representation Learning for Offline Meta-Reinforcement Learning with Data Limitations (AAAI 2024 · similarity 0.93)
- Decoupling Meta-Reinforcement Learning with Gaussian Task Contexts and Skills (AAAI 2024 · similarity 0.91)
- MetaCARD: Meta-Reinforcement Learning with Task Uncertainty Feedback via Decoupled Context-Aware Reward and Dynamics Components (AAAI 2024 · similarity 0.90)
- CERTAIN: Context Uncertainty-aware One-Shot Adaptation for Context-based Offline Meta Reinforcement Learning (ICML 2025 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.