Research map / Multi-agent systems and games / Human-AI and goal-conditioned
Quantifying Misalignment Between Agents: Towards a Sociotechnical Understanding of Alignment
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Adapts a computational social science model of contention into a probabilistic, goal-based misalignment score for mixed human/AI agent populations across problem areas, validated by simulations and two case studies (shopping recommender, CARLA autonomous vehicle).
Problem
Existing alignment work is qualitative, binary, focused on single agents or humanity as a monolith, and lacks a way to quantify misalignment among many human and AI agents with conflicting goals across problem areas.
Approach
Measures misalignment as the probability that two randomly sampled agents hold conflicting goals in a problem area, weighted by agents' importance weights and averaged across problem areas; extends a contention model from computational social science to goals, mixed agent populations, and problem areas.
Key results
- Shopping recommender case: misalignment 0.09 for food area, 0.42 for household area, 0.26 overall.
- Misalignment converges to (|G|-1)/|G| as uniformly distributed population grows, validated in simulations across goal counts.
- In the AV case study with 1000 agents, low weights across many problem areas keep misalignment low at almost all times even with full conflict.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Increase the number of mutually exclusive goals per problem area | Misalignment model · Simulated populations | misalignment | Helped |
| Reduce the weight of one goal group in a problem area | Misalignment model · Weight sensitivity simulation | misalignment: 0.75 | Hurt |
| Distribute agents evenly among conflicting goals | Misalignment model · Goal distribution simulation | misalignment | Helped |
| Increase the conflict level between goals | Misalignment model · Conflict levels simulation | misalignment | Helped |
| Vary how conflict is distributed across problem areas while keeping average conflict the same | Misalignment model · Conflict levels simulation | misalignment | No clear effect |
Surprising findings
- Misalignment is low in the AV case largely because of low pedestrian weights across many problem areas, even when conflict is maximal.
Limitations
- Assumes static snapshot of goals and no synergies, hierarchy, or power dynamics between agents.
- Assumes each agent holds one goal per problem area and agents are equally likely to be sampled.
- Case study weights and conflict values are estimated/assumed rather than empirically derived.
Experimental setup
Datasets: CARLA challenge scenario 17 (penalty coefficients)
Metrics: misalignment score (probability of conflict between random a
Builds on: computational model of contention in human populations
Closest papers by meaning
- Position: Towards Bidirectional Human-AI Alignment (NeurIPS 2025 · similarity 0.92)
- An Evolutionary Perspective on AI Alignment (Student Abstract) (AAAI 2025 · similarity 0.89)
- Measuring Error Alignment for Decision-Making Systems (AAAI 2025 · similarity 0.88)
- AI Alignment with Changing and Influenceable Reward Functions (ICML 2024 · similarity 0.88)
- Position: Towards Unified Alignment Between Agents, Humans, and Environment (ICML 2024 · similarity 0.87)
- Goal Alignment: Re-analyzing Value Alignment Problems Using Human-Aware AI (AAAI 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.