Research map / Multi-agent systems and games / Human-AI and goal-conditioned
Reward Design for Justifiable Sequential Decision-Making
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes a debate-based reward model in which two argumentative agents propose evidence for competing sepsis treatment decisions and a learned judge proxy scores justifiability, mixed with environment reward to train justifiable RL policies.
Problem
Reward specification for RL agents that should make decisions which can be justified with supporting evidence in accountable, high-stakes settings such as septic patient treatment.
Approach
Model the justifiability of a decision as the value of a two-player zero-sum debate game in which agents propose state-feature evidence; a learned Bradley-Terry judge scores evidence, and the game value is mixed into the RL reward via a debate coefficient lambda, with contextualized argumentative policies trained by PPO (self-play or maxmin).
Key results
- Judge accuracy on preferred action rises from 65% alone to nearly 90% when debate agents propose 6 evidence.
- Against an isolated agent, a confuser drops judge accuracy to 38%, while maxmin and self-play agents keep it near 85%.
- Debate-based feedback achieves alignment comparable to state-based feedback while using about 13.6% of the state (6 of 44 features).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use debate-generated evidence instead of uniform-random evidence | Argumentative agents and the judge · Sepsis preference recovery, L=6 | judge accuracy: 65% to a near 90% | Helped |
| Add multi-agent debate rather than proposing evidence in isolation (ablation) | Argumentative agents evaluated against a confuser · Sepsis preference recovery with confuser, L=6 | judge accuracy: 38\% to about 85\% | Helped |
| Use SHAP-ranked state features instead of argumentative evidence | Evidence used to justify decisions · Sepsis preference recovery | judge accuracy: about $70\%$ versus near $90\%$ | Hurt |
| Limit debate evidence from L=6 to L=4 (ablation) | Debate-based reward and argumentative agents · Sepsis policy performance | WIS | No clear effect |
| Limit debate evidence from L=6 to L=4 (ablation) | Argumentative agents · Sepsis preference recovery | judge accuracy: almost 90% in both settings | No clear effect |
Surprising findings
- The isolated argumentative agent is highly vulnerable to refutation, with a confuser flipping judge accuracy to 38%.
- Debate-based feedback using about 13.6% of the state performs comparably to full-state feedback.
Limitations
- Preferences are synthetic: clinician actions are assumed always justified and the judge reaches only 65% accuracy on test preferences.
- Arguments are restricted to predefined state features; no human judges were used.
- The performance-justifiability trade-off requires careful tuning of the debate coefficient lambda.
Experimental setup
Datasets: MIMIC-III v1.4
Benchmarks: WIS policy evaluation on held-out sepsis test set
Metrics: weighted importance sampling value fraction preferred to baseline by judge judge accuracy in recovering preference fraction of aligned decisions mortality vs dose deviation
Baselines: clinician policy baseline policy (lambda=0) state-based feedback judge isolated argumentative agent SHAP feature attribution
Builds on: Debate (Irving et al.) Double DQN with dueling architecture Prioritized experience replay PPO Bradley-Terry model
Closest papers by meaning
- When Debate Helps: Proposal Supply and Verification-Aware Readout in Multi-Agent Reasoning (NeurIPS 2026 · similarity 0.85)
- Epistemic Gain, Aleatoric Cost: Uncertainty Decomposition in Multi-Agent Debate for Math Reasoning (ICML 2026 · similarity 0.84)
- Debate or Vote: Which Yields Better Decisions in Multi-Agent Large Language Models? (NeurIPS 2025 · similarity 0.84)
- Multi-LLM Debate: Framework, Principals, and Interventions (NeurIPS 2024 · similarity 0.84)
- Capturing Individual Human Preferences with Reward Features (NeurIPS 2025 · similarity 0.84)
- Towards Theoretical Understanding of Sequential Decision Making with Preference Feedback (ICML 2025 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.