Research map / Reinforcement learning / RL from feedback
Meta-Reinforcement Learning with Adaptation from Human Feedback via Preference-Order-Preserving Task Embedding
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes POEM, a meta-RL method that learns a preference-order-preserving task embedding from reward-based meta-training so that few-shot human preference queries can infer task embeddings and task-specific policies.
Problem
Meta-RL with adaptation from human feedback: adapting a pre-trained meta-model to new tasks using few preference queries and few trajectories, without reward signals at meta-test time.
Approach
Trains a PEARL-style context encoder with an asymmetric similarity (reward embedding vs. policy embedding inner product) so that task-embedding similarity order matches the ordering of cross-task policy performance; at adaptation, candidate embeddings are filtered and refined using pairwise preference queries over policies.
Key results
- On HalfCheetah-Vel with 10 preference queries, POEM reaches -46±6 versus -112 to -182 for baselines (ANOLE -112±14, MAML-reward -182±11).
- On Ant-Dir-2D with 10 queries, POEM reaches 749±40 versus 588±67 for ANOLE, per the paper's table.
- The paper reports 20%-50% improvement over the best baselines across scenarios, and POEM remains robust under a Boltzmann-noise preference oracle.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use POEM instead of the strongest listed baseline, ANOLE | HalfCheetah-Vel · HalfCheetah-Vel, 10 queries | average accumulated reward: -112±14 -> -46±6 | Helped |
| Use POEM instead of ANOLE-greedy | HalfCheetah-Fwd-Back · HalfCheetah-Fwd-Back, 10 queries | average accumulated reward: 1493±89 -> 1709±23 | Helped |
| Use POEM instead of ANOLE-random | Ant-Goal · Ant-Goal, 10 queries | average accumulated reward: -378±16 -> -202±7 | Helped |
| Use POEM instead of ANOLE-greedy | Ant-Fwd-Back · Ant-Fwd-Back, 10 queries | average accumulated reward: 820±32 -> 1015±48 | Helped |
| Use POEM instead of ANOLE | Ant-Dir-2D · Ant-Dir-2D, 10 queries | average accumulated reward: 588±67 -> 749±40 | Helped |
| Use POEM instead of ANOLE-greedy | Ant-Vel · Ant-Vel, 10 queries | average accumulated reward: 110±5 -> 112±4 | Helped |
| Add separate reward and policy embedding encoders with the asymmetric similarity metric (ablation) | POEM · HalfCheetah-Vel meta-training | average accumulated reward | Helped |
| Use POEM’s context design instead of the PEARL context (ablation) | POEM · HalfCheetah-Vel meta-training | average accumulated reward | Helped |
Limitations
- Theoretical guarantee relies on strong assumptions (encoder well-trained, bijective reward embedding map, optimal decoder policy)
- Meta-training uses ground-truth rewards and a simulated preference oracle, not real humans
- Evaluated only on continuous-control simulation benchmarks
Experimental setup
Datasets: MuJoCo locomotion tasks (HalfCheetah, Ant, Humanoid) MetaWorld ML1 and ML10
Benchmarks: HalfCheetah-Vel HalfCheetah-Fwd-Back Ant-Vel Ant-Fwd-Back Ant-Dir-2D Ant-Goal Humanoid-Dir-2D MetaWorld-ML1 MetaWorld-ML10
Metrics: average accumulated reward on test tasks accumulated reward vs. number of preference queries
Baselines: ANOLE ANOLE-random ANOLE-greedy MAML-reward oracle (reward-based meta-test)
Builds on: PEARL ANOLE MAML soft actor-critic Bradley-Terry model
Closest papers by meaning
- MetaCARD: Meta-Reinforcement Learning with Task Uncertainty Feedback via Decoupled Context-Aware Reward and Dynamics Components (AAAI 2024 · similarity 0.90)
- Alleviating Shifted Distribution in Human Preference Alignment through Meta-Learning (AAAI 2025 · similarity 0.89)
- Meta-Reinforcement Learning with Universal Policy Adaptation: Provable Near-Optimality under All-task Optimum Comparator (NeurIPS 2024 · similarity 0.89)
- AMAGO-2: Breaking the Multi-Task Barrier in Meta-Reinforcement Learning with Transformers (NeurIPS 2024 · similarity 0.88)
- Generalizable Task Representation Learning for Offline Meta-Reinforcement Learning with Data Limitations (AAAI 2024 · similarity 0.87)
- Capturing Individual Human Preferences with Reward Features (NeurIPS 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.