Research map / Reinforcement learning / Visual and representation RL
Generalizable Task Representation Learning for Offline Meta-Reinforcement Learning with Data Limitations
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes GENTLE, an offline meta-RL method that learns task representations with a reconstruction-based Task Auto-Encoder plus policy/dynamics/reward relabeling of probing data, and uses TD3+BC as the meta-policy backbone.
Problem
Offline meta-RL task representation learning overfits and fails to generalize when training tasks are few and behavior data lacks diversity.
Approach
Replace contrastive task-representation objectives with a Task Auto-Encoder trained to reconstruct next states and rewards from probing data, and align the probing distribution with test-time distribution by relabeling states with the current meta-policy actions and pretrained dynamics/reward models across tasks.
Key results
- Under the one-shot protocol on Ant-Dir test tasks, GENTLE reaches 464.11 return versus 103.20 for FOCAL and 63.08 for BOReL.
- Under the given-context protocol on Hopper-Params test tasks, GENTLE achieves 251.60 vs 224.56 for FOCAL, 125.75 for CORRO and 149.74 for BOReL.
- Replacing the TAE objective with a contrastive objective (GENTLE-Contrastive) lowers one-shot Ant-Dir test return from 464.11 to 376.37 in the ablation.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use reconstruction training for TAE instead of a contrastive objective (ablation) | GENTLE · Ant-Dir Test (one-shot) | return: 376.37±110.18 -> 464.11±95.74 | Helped |
| add relabeling to construct pseudo-transitions for TAE training (ablation) | GENTLE · Ant-Dir Test (one-shot) | return: 125.99±61.42 -> 464.11±95.74 | Helped |
| add policy relabeling instead of using dataset actions (ablation) | GENTLE · Ant-Dir Test (one-shot) | return: 356.20±75.99 -> 464.11±95.74 | Helped |
| increase sampling from other task datasets from a 1:0 ego-to-other ratio (ablation) | GENTLE · Ant-Dir Test | return | Helped |
| increase sampling ratios beyond 1:9, or use ratios 1:12 and 1:15 (ablation) | GENTLE · Ant-Dir Test | return | Mixed |
| increase the number of training tasks (ablation) | GENTLE · Ant-Dir Test | return | Helped |
| replace expert-only context data with mixed-policy context data | FOCAL · Ant-Dir Test (one-shot) | return: 103.20±56.39 -> 213.48±81.40 | Helped |
| replace expert-only context data with mixed-policy context data | GENTLE · Ant-Dir Test (one-shot) | return: 464.11±95.74 -> 481.10±80.07 | No clear effect |
Surprising findings
- GENTLE's representation quality stays nearly unchanged when behavior context shifts from expert to mixed data, while FOCAL improves markedly with more diverse context.
- Original baselines with their native backbones (BRAC/SAC) performed worse than the TD3+BC re-implementations in the authors' runs.
Limitations
- Does not address sparse reward settings.
- Does not tackle exploration policy development during meta-testing.
- Performance degrades when the number of training tasks is small (4 tasks) and relabeling with estimated dynamics models may be erroneous on out-of-task states.
Experimental setup
Datasets: offline datasets collected by SAC expert policies on Point-Robot, Ant-Dir, Cheetah-Vel, Cheetah-Dir, Hopper-Params, Walk medium and mixed behavior context data (Ant-Dir)
Benchmarks: Point-Robot Ant-Dir Cheetah-Vel Cheetah-Dir Hopper-Params Walker-Params
Metrics: average return of last checkpoint over 8 seeds
Baselines: FOCAL CORRO BOReL MACAW GENTLE-Contrastive
Builds on: TD3+BC FOCAL CORRO BOReL PEARL
Closest papers by meaning
- Scrutinize What We Ignore: Reining In Task Representation Shift Of Context-Based Offline Meta Reinforcement Learning (ICLR 2025 · similarity 0.93)
- Entropy Regularized Task Representation Learning for Offline Meta-Reinforcement Learning (AAAI 2025 · similarity 0.93)
- Towards an Information Theoretic Framework of Context-Based Offline Meta-Reinforcement Learning (NeurIPS 2024 · similarity 0.92)
- Decoupling Meta-Reinforcement Learning with Gaussian Task Contexts and Skills (AAAI 2024 · similarity 0.90)
- Task-Aware Virtual Training: Enhancing Generalization in Meta-Reinforcement Learning for Out-of-Distribution Tasks (ICML 2025 · similarity 0.90)
- Offline Multitask Representation Learning for Reinforcement Learning (NeurIPS 2024 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.