Research map / Multi-agent systems and games / Human-AI and goal-conditioned
Goal Alignment: Re-analyzing Value Alignment Problems Using Human-Aware AI
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Reformulates value alignment as human-aware goal alignment in deterministic planning, and proposes an interactive goal-elicitation algorithm that uses the human's plan under their (possibly incorrect) model of the robot to query which fluents belong to the true goal.
Problem
Value alignment work focuses on complexity of reward-based objective specification and ignores the asymmetry between the human's beliefs about the agent model and the agent's true model, which causes goal misspecification.
Approach
Treats misspecification as arising from human-agent model asymmetry: uses the human's plan simulated in the human's model as an upper bound on the true goal, and uses an approximate value-of-information query strategy (with goal-recognition-style probabilities) to ask about fluents, aiming to find a plan that satisfies the hidden goal with few queries.
Key results
- Across five IPC domains with five instances each, the algorithm's average query count was far below the naive upper bound |S^H_G* \ G_H| in most domains (e.g., Driverlog, Elevators, Logistics often 0 queries).
- Blocksworld showed the smallest gains, with e.g. 6.4 queries on average versus an upper bound of 7.
- On the running tea-making example, the algorithm averaged 4.2 queries (max 8) over ten runs.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use the proposed goal-alignment algorithm instead of querying every potential goal predicate | Goal alignment algorithm · Blocksworld | number of queries: 7 -> 6.4 | Helped |
| Use the proposed goal-alignment algorithm instead of querying every potential goal predicate | Goal alignment algorithm · Driverlog | number of queries: 21 -> 0 | Helped |
| Use the proposed goal-alignment algorithm instead of querying every potential goal predicate | Goal alignment algorithm · Elevators | number of queries: 25 -> 0 | Helped |
| Use the proposed goal-alignment algorithm instead of querying every potential goal predicate | Goal alignment algorithm · Logistics | number of queries: 12 -> 10.8 | Helped |
| Use the proposed goal-alignment algorithm instead of querying every potential goal predicate | Goal alignment algorithm · Rover | number of queries: 46 -> 0 | Helped |
Surprising findings
- For many problems the algorithm needed zero queries to find a plan guaranteed to satisfy the hidden human goal, by achieving a superset of the goal state expected by the human.
Limitations
- Restricted to deterministic goal-directed classical planning; extension to decision-theoretic settings left to future work.
- Assumes access to the human's model of the robot (DH and IH) and that human and robot share fluents.
- Blocksworld shows smaller gains, and the approximation of the value function is a lower bound rather than exact.
Experimental setup
Datasets: IPC benchmark problems (Blocksworld, Driverlog, Elevators, Rover, Logistics)
Benchmarks: IPC Blocksworld IPC Driverlog IPC Elevators IPC Rover IPC Logistics
Metrics: number of queries computation time (secs)
Baselines: naive query-all-fluents upper bound
Builds on: Human-Aware AI Assistive games (Hadfield-Menell et al. 2016) goal recognition (Ramírez and Geffner 2010) explicable planning
Closest papers by meaning
- Expectation Alignment: Handling Reward Misspecification in the Presence of Expectation Mismatch (NeurIPS 2024 · similarity 0.90)
- Position: Towards Bidirectional Human-AI Alignment (NeurIPS 2025 · similarity 0.87)
- Quantifying Misalignment Between Agents: Towards a Sociotechnical Understanding of Alignment (AAAI 2025 · similarity 0.86)
- An Evolutionary Perspective on AI Alignment (Student Abstract) (AAAI 2025 · similarity 0.86)
- AI Alignment with Changing and Influenceable Reward Functions (ICML 2024 · similarity 0.85)
- Measuring Error Alignment for Decision-Making Systems (AAAI 2025 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.