Research map / Bandits and online learning / Multi-armed bandits
Enhancing Preference-based Linear Bandits via Human Response Time
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes a linear-regression utility estimator combining binary choices and response times under the difference-based EZ diffusion model, and plugs it into generalized successive elimination for fixed-budget preference-based best-arm identification.
Problem
Preference learning from binary choices gives little information about preference strength for strongly preferred queries, and existing choice+response-time models are computationally expensive for interactive systems.
Approach
Use the dEZDM identity E[c_x]/E[t_x] = x^T(theta/a) to turn choices and decision times into a continuous signal, estimated by OLS across queries; this extracts information from strongly preferred queries where choice-only estimators saturate, and is integrated into GSE for fixed-budget best-arm identification.
Key results
- On three real-world human choice and response time datasets, the choice-decision-time GSE variant consistently outperforms the choice-only GSE variant in fixed-budget best-arm identification error.
- Theory: asymptotic weight for choice-decision-time estimator (min expected decision time) stays large for strong preferences whereas choice-only weight a^2 Var(c_x) decays to zero.
- Choice-decision-time estimator performs similarly when non-decision time is unknown (using response time instead of decision time).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add decision times to choices using the choice-decision-time estimator | GSE with transductive design · Three real-world datasets | best-arm identification error | Helped |
| Use the choice-decision-time estimator instead of the choice-only estimator | Utility estimation · Synthetic sphere problem, strong preferences | best-arm identification error | Helped |
| Use the choice-decision-time estimator instead of the choice-only estimator | Utility estimation · Synthetic sphere problem, weak preferences | best-arm identification error | No clear effect |
| Use the choice-only estimator with weak-preference design instead of transductive design (ablation) | GSE with choice-only estimator · Synthetic sphere problem, moderate preference scaling | best-arm identification error | Helped |
| Use the choice-only estimator with weak-preference design instead of transductive design (ablation) | GSE with choice-only estimator · Synthetic sphere problem, very strong preferences | best-arm identification error | Hurt |
| Use the choice-only estimator with weak-preference design instead of transductive design (ablation) | GSE with choice-only estimator · Real-world datasets | best-arm identification error | Hurt |
| Use response times in place of decision times when non-decision time is unknown (ablation) | GSE with choice-decision-time estimator · Three real-world datasets | best-arm identification error | No clear effect |
| Replace the choice-decision-time estimator with the choice-only logit estimator (ablation) | GSE with transductive design · Three real-world datasets | best-arm identification error | Hurt |
| Replace the choice-decision-time estimator with the choice-decision-time logit estimator (ablation) | GSE with transductive design · Three real-world datasets | best-arm identification error | Hurt |
| Increase the barrier parameter | Utility estimation · Synthetic sphere problem | best-arm identification error | Helped |
Surprising findings
- Weak-preference experimental design does not outperform the transductive design in practice, since weak-preference queries take longer to answer and θ* is unknown.
Limitations
- Relies on reliable response time data, which may be hard to obtain in crowdsourcing settings.
- Assumes non-decision time is known (though an unknown-case variant is evaluated empirically).
- Random resource usage requires manual tuning of elimination parameter eta and buffer size B_buff.
Experimental setup
Datasets: Food-risk dataset Snack dataset
Benchmarks: sphere synthetic problem Food-risk dataset Snack dataset
Metrics: best-arm identification error probability probability of incorrect argmax estimate
Baselines: choice-only estimator (logistic MLE) transductive design with choice-only estimator weak-preference design choice-decision-time logit estimator choice-only logit estimator
Builds on: EZ diffusion model Generalized Successive Elimination transductive linear logistic bandit logistic regression MLE
Closest papers by meaning
- Preference Learning with Response Time: Robust Losses and Guarantees (NeurIPS 2025 · similarity 0.92)
- ResponseRank: Data-Efficient Reward Modeling through Preference Strength Learning (NeurIPS 2025 · similarity 0.90)
- Preference Learning of Latent Decision Utilities with a Human-like Model of Preferential Choice (NeurIPS 2024 · similarity 0.89)
- Capturing Individual Human Preferences with Reward Features (NeurIPS 2025 · similarity 0.88)
- Neural Dueling Bandits: Preference-Based Optimization with Human Feedback (ICLR 2025 · similarity 0.88)
- DOPL: Direct Online Preference Learning for Restless Bandits with Preference Feedback (ICLR 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.