Research map / Reinforcement learning / MDPs and planning
Expectation Alignment: Handling Reward Misspecification in the Presence of Expectation Mismatch
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Formalizes reward misspecification as expectation mismatch between human and robot MDP models and proposes an LP-based interactive query algorithm to find expectation-aligned policies in grid-world MDPs.
Problem
Reward functions specified by users may be misspecified because the user's beliefs about the agent's model differ from reality, so optimal policies can violate the user's expectations; no clear formal definition of misspecification or of resolving it existed.
Approach
Model the human's reward specification as derived from behavioral expectations (occupancy-frequency constraints on forbidden and goal states) under the human's domain model, define misspecification and expectation-aligned policies, and compute supersets of forbidden/goal states via LPs, querying the user on ambiguous states until an aligned policy is found.
Key results
- On the largest grids the proposed method ran in under five minutes with only a few queries, while MMRQ-k failed to solve any instance within the 30-minute limit.
- IRD produced policies violating expectations in almost all settings (e.g., 3.6 violated expectations on 11x11 Walkway), whereas the proposed method guarantees no violations.
- Query counts stayed far below the state-space size across all five domains.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use interactive expectation-alignment queries instead of modified IRD's reward-based planning | Expectation Alignment versus modified IRD · Walkway, Obstacles, Four Rooms, Puddle, and Maze | expectation violations | Helped |
| Use Expectation Alignment instead of MMRQ-k (k=2) | MDP benchmark instances · Five benchmark domains | solved instances | Helped |
Surprising findings
- Minimax-Regret Querying could not solve any instance because computing the exponential set of dominant policies dominated runtime.
Limitations
- Only considers human-specified forbidden/goal state expectations under known models and planning functions.
- Does not address errors in transcription or communication of the reward function.
- Assumes a specific expectation-set form; other expectation forms are not explored.
Experimental setup
Datasets: SimpleRL grid-world benchmarks (Walkway, Obstacles, Four Rooms, Puddle, Maze)
Benchmarks: Walkway Obstacles Four Rooms Puddle Maze
Metrics: query count runtime (seconds) number of violated expectations
Baselines: MMRQ-k Inverse Reward Design (modified)
Builds on: Minimax-Regret Querying Inverse Reward Design CIRL
Closest papers by meaning
- Goal Alignment: Re-analyzing Value Alignment Problems Using Human-Aware AI (AAAI 2024 · similarity 0.90)
- AI Alignment with Changing and Influenceable Reward Functions (ICML 2024 · similarity 0.87)
- Measuring Error Alignment for Decision-Making Systems (AAAI 2025 · similarity 0.87)
- Learning to Decide with AI Assistance under Human-Alignment (NeurIPS 2026 · similarity 0.85)
- An Evolutionary Perspective on AI Alignment (Student Abstract) (AAAI 2025 · similarity 0.85)
- Model Alignment as Prospect Theoretic Optimization (ICML 2024 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.