Research map / Learning theory and algorithms / Submodular and game algorithms
The Burden of Interactive Alignment with Inconsistent Preferences
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Models user engagement as a System 2 (rational engage/disengage) and System 1 (impulsive duration) split, and analyzes a multi-leader single-follower Stackelberg game in which users lead and the algorithm best-responds, defining a critical foresight horizon for alignment and showing costly signals reduce it.
Problem
Users with inconsistent preferences spend time on content that offers little long-term value, which misleads engagement-maximizing algorithms; the question is how far-sighted a user must be to steer the algorithm toward her true interests.
Approach
Users commit to engagement strategies as Stackelberg leaders; the algorithm (follower) best-responds via a linear classifier on its posterior over user types. Alignment requires the user's effective horizon 1/(1-gamma) to exceed a type-specific threshold, and an observable costly signal (e.g., an extra click) lowers that threshold.
Key results
- Type 1 users achieve constant regret only if effective horizon 1/(1-gamma) exceeds r(b)/(r(b)-r(a)), i.e., a critical horizon exists.
- Costly signaling reduces the required effective horizon by r(a)alpha(a)/(r(b)alpha(b)) - c/r(b) (in the stated relative terms).
- Algorithm best response is a linear classifier over the posterior, with weights depending only on each type's own strategy.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use an effective horizon below the type-1 threshold | type-1 users without costly signaling · interactive alignment equilibrium | alignment | Hurt |
| add an observable costly signal | type-1 users with inconsistent preferences · algorithmic-entry equilibrium | required effective horizon | Helped |
| add observable costly signaling when the steerable set is empty | sufficiently foresighted type-1 users · algorithmic-entry equilibrium | engagement with undesired content | Helped |
Surprising findings
- Users who are not sufficiently foresighted are effectively aligned to the algorithm's engagement objective despite leading the game.
Limitations
- Assumes users know their rewards and the algorithm knows the user strategies (full information).
- Two-item special case with finite user types and a simplified geometric engagement model.
- Only a two-sided user-platform interaction; content creators are not modeled.
Experimental setup
Metrics: regret effective horizon threshold
Builds on: strategic classification Bellman update
Closest papers by meaning
- Direct Alignment with Heterogeneous Preferences (NeurIPS 2025 · similarity 0.88)
- Understanding the Learning Dynamics of Alignment with Human Feedback (ICML 2024 · similarity 0.88)
- AI Alignment with Changing and Influenceable Reward Functions (ICML 2024 · similarity 0.87)
- Bounded Rationality for LLMs: Satisficing Alignment at Inference-Time (ICML 2025 · similarity 0.86)
- AlignDiff: Aligning Diverse Human Preferences via Behavior-Customisable Diffusion Model (ICLR 2024 · similarity 0.86)
- Quantifying Misalignment Between Agents: Towards a Sociotechnical Understanding of Alignment (AAAI 2025 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.