Research map / Reinforcement learning / Visual and representation RL
Learning to Make Adherence-aware Advice
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Models human-AI advice-giving as an MDP where the human's adherence level and a defer option let the machine advise only at critical states, and gives UCB and reward-free exploration algorithms with improved sample complexity over problem-agnostic RL.
Problem
Human-AI decision support where humans may ignore AI advice (unknown adherence level) and AI should advise selectively rather than at every step.
Approach
Model the human as following a suboptimal policy with a probabilistic adherence level, augment the machine's action space with a defer option, and exploit the structure (monotonicity in adherence, reward-free exploration) to derive tailored learning algorithms with tighter sample complexity bounds than generic RL, plus a penalty beta (or CMDP budget D) to make advice pertinent.
Key results
- Theory: UCB-AD achieves O(H^2S^2A/eps^2) PAC sample complexity under known transitions, improving over O(H^3S^2A/eps^2) from prior model-based analysis and O(H^4SA/eps^2) from model-free analysis.
- RFE-beta learns near-optimal pertinent-advice policies for all beta in [0,H) simultaneously with O(H^5SA/eps^2) episodes, and RFE-CMDP does so for all budgets D.
- In Flappy Bird, UCB-AD achieves lower regret than RFE-AD and EULER, and lower regret with higher adherence (theta=0.8 vs 0.4).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use the adherence-aware UCB-AD algorithm instead of problem-agnostic RL | Learning an advice policy in Environment E1 · Flappy Bird | regret | Helped |
| Increase adherence from θ(a,s)≡0.4 to θ(a,s)≡0.8 | UCB-AD · Flappy Bird, Policy Greedy and Policy Safe | regret | Helped |
| Use RFE-CMDP instead of UC-CFH | Advice-budget-constrained learning · Flappy Bird, Policy Safe, D = 1 | convergence rate | Helped |
| Use the transition kernel learned with advice budget D = 1 to produce policies for other budgets | RFE-CMDP · Flappy Bird, advice budgets D = 2, 3, 4, and 5 | near-optimal policy performance | Helped |
| Use one exploration phase to support policies for all advice budgets rather than rerunning for each budget | RFE-CMDP compared with UC-CFH · Flappy Bird, multiple advice budgets | sample efficiency | Helped |
Limitations
- Experiments are limited to small grid-based game environments (Flappy Bird, car driving) with simulated human policies.
- Assumes the human's policy and adherence level follow a fixed stationary model; analysis for the known-environment setting relies on known transitions.
Experimental setup
Benchmarks: Flappy Bird grid environment Car Driving environment
Metrics: regret value gap
Baselines: EULER UC-CFH
Builds on: UCB-based episodic RL (EULER) RF-Express reward-free exploration UC-CFH constrained MDP algorithm
Closest papers by meaning
- Learning Personalized Decision Support Policies (AAAI 2025 · similarity 0.86)
- Towards Theoretical Understanding of Sequential Decision Making with Preference Feedback (ICML 2025 · similarity 0.85)
- Utilizing Human Behavior Modeling to Manipulate Explanations in AI-Assisted Decision Making: The Good, the Bad, and the Scary (NeurIPS 2024 · similarity 0.84)
- Learning to Decide with AI Assistance under Human-Alignment (NeurIPS 2026 · similarity 0.84)
- Strategyproof Reinforcement Learning from Human Feedback (NeurIPS 2025 · similarity 0.84)
- GUIDE: Real-Time Human-Shaped Agents (NeurIPS 2024 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.