Research map / Reinforcement learning / Visual and representation RL
Learning Personalized Decision Support Policies
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Modiste, an interactive tool that learns a personalized decision support policy online per decision-maker using contextual bandits (LinUCB and KNN), and validates it in simulations and with 80 Prolific participants on CIFAR and MMLU.
Problem
Deciding, for each individual decision-maker with no prior data, which form of decision support (e.g., model prediction, expert consensus, LLM output, or no support) to show on each input so that their accuracy is maximized.
Approach
Frame decision support selection as a stochastic contextual bandit where support forms are arms and the human's prediction error under each form is learned online, yielding a per-human policy that picks argmin estimated error rather than requiring offline human decision data.
Key results
- In the CIFAR-3A human study, KNN-based Modiste achieved significantly lower expected loss than all offline variants (p<0.001 across pairs) for participants with varying expertise.
- In simulation on varying expertise profiles, Modiste-KNN excess loss was 0.08 vs 0.15 for population-wide and 0.50 for H-Only on CIFAR-3A.
- On MMLU-2A with mostly strictly-better profiles, Modiste variants matched the H-LLM baseline on average while having smaller variance; both outperformed H-Only (p<0.01).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use personalized Modiste-KNN instead of offline support policies | Decision support policy · CIFAR-3A varying expertise | excess loss: 0.08 ± 0.02 | Helped |
| Use personalized Modiste-LinUCB instead of the fixed Human Alone policy | Decision support policy · CIFAR-3A varying expertise | excess loss: 0.50 ± 0.06; 0.19 ± 0.05 | Hurt |
| Personalize support rather than use a fixed form of support | Decision support policy · CIFAR-3A invariant expertise | excess loss: 0.00 ± 0.01 | No clear effect |
| Use personalized policies rather than offline policies | Decision support policy · CIFAR-3A strictly better expertise | excess loss: 0.06 ± 0.01 | No clear effect |
| Use personalized Modiste-KNN rather than offline policies | Decision support policy · MMLU-2A varying expertise | excess loss: 0.05 ± 0.03 | Helped |
| Use personalized policies rather than Human Alone | Decision support policy · MMLU-2A strictly better expertise | excess loss: p<0.01 | Helped |
| Use personalized policies rather than always provide LLM support | Decision support policy · MMLU-2A strictly better expertise | excess loss | No clear effect |
| Use Modiste-KNN instead of Modiste-LinUCB | Modiste · CIFAR-3A varying expertise | excess loss: 0.19 ± 0.05; 0.08 ± 0.02 | Helped |
| Use Modiste-KNN instead of Modiste-LinUCB | Modiste · MMLU-2A varying expertise | excess loss: 0.07 ± 0.04; 0.05 ± 0.03 | Helped |
| Personalize support rather than use a population-wide policy | Decision support policy · CIFAR-3A varying expertise | excess loss: 0.15 ± 0.03; 0.08 ± 0.02 | Helped |
Surprising findings
- KNN generally outperformed LinUCB, which is attributed to LinUCB's implicit linearity assumption.
- Most MMLU participants were better with LLM access, so personalization mainly matched the best fixed baseline rather than beating it.
Limitations
- Only classification with immediate feedback; does not address delayed feedback or planning/perception tasks.
- Requires domain experts to define the action set of forms of support.
- Risk of over-reliance on erroneous support is not addressed.
Experimental setup
Datasets: CIFAR-10 CIFAR-10H MMLU
Benchmarks: CIFAR-10 (5-class animal subset, CIFAR-3A) MMLU (4 topics, MMLU-2A)
Metrics: expected loss (excess loss over optimal) standard error statistical significance (ANOVA with Tukey HSD)
Baselines: Human Alone Human + Model Human + Consensus Human + LLM Population-wide majority vote policy
Builds on: LinUCB KNN-UCB learning to defer
Closest papers by meaning
- Personalized Reinforcement Learning with a Budget of Policies (AAAI 2024 · similarity 0.86)
- Learning to Make Adherence-aware Advice (ICLR 2024 · similarity 0.86)
- Decoding AI’s Nudge: A Unified Framework to Predict Human Behavior in AI-Assisted Decision Making (AAAI 2024 · similarity 0.85)
- Capturing Individual Human Preferences with Reward Features (NeurIPS 2025 · similarity 0.85)
- Learning to Decide with AI Assistance under Human-Alignment (NeurIPS 2026 · similarity 0.85)
- Personalizing Reinforcement Learning from Human Feedback with Variational Preference Learning (NeurIPS 2024 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.