Research map / Reinforcement learning / Visual and representation RL
Towards an Information Theoretic Framework of Context-Based Offline Meta-Reinforcement Learning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Unifies context-based offline meta-RL methods (FOCAL, CORRO, CSRO) under an information-theoretic bound on I(Z;M) and proposes UNICORN-SUP and UNICORN-SS, which outperform baselines on MuJoCo/MetaWorld OMRL benchmarks including OOD behavior-policy contexts.
Problem
Context-based offline meta-RL suffers from poor generalization under context shift (OOD behavior-policy contexts), and existing methods lack a unified theoretical account of their task-representation objectives.
Approach
Shows FOCAL, CORRO and CSRO optimize upper bound, lower bound, and interpolation of I(Z;M) respectively, and derives two instantiations: a supervised cross-entropy version and a self-supervised version combining a contrastive FOCAL term with a reconstruction term for I(Xt;Z,Xb).
Key results
- UNICORN-SS achieves the best IID return on HalfCheetah-Dir (1307±26 vs. 1180±228 for CSRO) and Walker-Param (419±44 vs. 310±58 for CSRO).
- Under OOD behavior-policy contexts, UNICORN-SS keeps 1296 on HalfCheetah-Dir versus 458 for CSRO and 245 for CORRO.
- On Ant-Dir with random-quality data, UNICORN-SS reaches 81±18 IID versus 2±3 for CSRO, though CSRO is slightly higher on mixed-data Ant-Dir IID (276 vs. 267).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use UNICORN-SS instead of FOCAL | COMRL · HalfCheetah-Dir IID | average return: 1186±272 -> 1307±26 | Helped |
| use UNICORN-SS instead of FOCAL | COMRL · HalfCheetah-Dir OOD | average return: 861±253 -> 1296±24 | Helped |
| use UNICORN-SS instead of FOCAL | COMRL · HalfCheetah-Vel IID | average return: -22±1 -> -22±1 | No clear effect |
| use UNICORN-SS instead of FOCAL | COMRL · HalfCheetah-Vel OOD | average return: -97±2 -> -94±5 | Helped |
| use UNICORN-SS instead of CSRO | COMRL · Ant-Dir IID | average return: 276±19 -> 267±14 | Hurt |
| use UNICORN-SS instead of CSRO | COMRL · Ant-Dir OOD | average return: 233±12 -> 236±18 | Helped |
| use UNICORN-SS instead of CSRO | COMRL · Hopper-Param IID | average return: 310±6 -> 316±6 | Helped |
| use UNICORN-SS instead of CSRO | COMRL · Hopper-Param OOD | average return: 301±10 -> 304±11 | Helped |
| use UNICORN-SS instead of CSRO | COMRL · Walker-Param IID | average return: 310±58 -> 419±44 | Helped |
| use UNICORN-SS instead of FOCAL | COMRL · Walker-Param OOD | average return: 286±91 -> 407±46 | Helped |
| use UNICORN-SS instead of CSRO | COMRL · Reach IID | average return: 2720±235 -> 2775±241 | Helped |
| use UNICORN-SS instead of CSRO | COMRL · Reach OOD | average return: 2801±182 -> 2604±183 | Hurt |
Surprising findings
- CORRO fails on Ant-Dir random, medium and expert datasets, which the authors attribute to its reliance on a negative sample generator.
- UNICORN-SUP underperforms UNICORN-SS, which the authors attribute to too few training tasks (n_M) for the supervised estimate.
Limitations
- Framework decouples task representation learning from downstream offline policy optimization, so it does not explain how representation quality affects downstream RL performance.
- Experiments are small in scale (at most 40 tasks and ~450k transitions), which may explain the weaker UNICORN-SUP results.
- Key derivations assume static M and X, which are violated in online RL, so the framework does not directly extend to online settings.
Experimental setup
Datasets: MuJoCo HalfCheetah-Dir MuJoCo HalfCheetah-Vel MuJoCo Ant-Dir MuJoCo Hopper-Param MuJoCo Walker-Param MetaWorld Reach
Benchmarks: HalfCheetah-Dir HalfCheetah-Vel Ant-Dir Hopper-Param Walker-Param Reach
Metrics: average testing return
Baselines: FOCAL CORRO CSRO Supervised MACAW Prompt-DT BOReL GENTLE
Builds on: FOCAL CORRO CSRO VariBAD BOReL
Closest papers by meaning
- Scrutinize What We Ignore: Reining In Task Representation Shift Of Context-Based Offline Meta Reinforcement Learning (ICLR 2025 · similarity 0.95)
- Entropy Regularized Task Representation Learning for Offline Meta-Reinforcement Learning (AAAI 2025 · similarity 0.93)
- Generalizable Task Representation Learning for Offline Meta-Reinforcement Learning with Data Limitations (AAAI 2024 · similarity 0.92)
- Pessimistic Latent Task-aware Optimization for Robust Offline Meta-Reinforcement Learning (NeurIPS 2026 · similarity 0.91)
- CERTAIN: Context Uncertainty-aware One-Shot Adaptation for Context-based Offline Meta Reinforcement Learning (ICML 2025 · similarity 0.91)
- Decoupling Meta-Reinforcement Learning with Gaussian Task Contexts and Skills (AAAI 2024 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.