Research map / Bandits and online learning / Contextual bandits and Thompson sampling
Direct Preference-Based Evolutionary Multi-Objective Optimization with Dueling Bandits
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes D-PBEMO, a preference-based evolutionary multi-objective optimizer that learns DM preferences directly via a clustering-based stochastic dueling bandit and converts them into a probabilistic preference distribution for EMO, with a KL-based stopping rule.
Problem
Preference-based evolutionary multi-objective optimization (PBEMO) is inefficient and misaligned with decision-maker (DM) preferences when using reward/model-based preference learning or when many pairwise queries are needed under stochastic human feedback; termination of DM consultation is also heu
Approach
Cluster the EMO population into K subsets treated as arms of a stochastic dueling bandit (double Thompson sampling at subset level), then convert the pairwise win/loss counts into a density-ratio-based Gaussian mixture preference distribution that plugs into NSGA-II and MOEA/D, with KL divergence between consecutive consultations as the stopping criterion.
Key results
- D-PBNSGA-II and D-PBMOEA/D achieved the best metric values in 77 of 96 comparisons (Wilcoxon rank-sum, 0.05 level) on synthetic problems.
- The advantage grows with number of objectives (m>=3), per the authors.
- Consultation converges in fewer than 10 iterations with epsilon=1e-3 according to the authors.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use D-PBNSGA-II or D-PBMOEA/D instead of peer PBEMO algorithms | PBEMO · 33 synthetic test problems | approximation accuracy ε* and average accuracy ε̄: 77 out of 96 comparisons | Helped |
| Use D-PBEMO instead of peer PBEMO algorithms | Protein structure prediction · PSP problems | RMSD | Helped |
| Use D-PBEMO instead of peer PBEMO algorithms | RNA inverse design · RNA inverse design tasks | target structure match | Helped |
| Replace clustering-based dueling bandits with double Thompson sampling (ablation) | D-PBEMO · Synthetic test problems | approximation accuracy ε* and average accuracy ε̄ | Hurt |
| Replace clustering-based dueling bandits with parameterized preference learning (ablation) | D-PBEMO · Problems with m=2 versus higher-dimensional problems | approximation accuracy ε* and average accuracy ε̄ | Mixed |
| Vary the number of clusters K among 2, 5, and 10 (ablation) | D-PBEMO · Synthetic test problems | approximation accuracy ε* and average accuracy ε̄ | No clear effect |
| Vary the consultation termination threshold ε among 10⁻¹, 10⁻³, and 10⁻⁶ (ablation) | D-PBEMO · Synthetic test problems | approximation accuracy ε* and average accuracy ε̄ | No clear effect |
| Set the consultation termination threshold to ε=10⁻¹ (ablation) | D-PBEMO · Synthetic test problems | premature convergence risk | Hurt |
| Set the consultation termination threshold to ε=10⁻⁶ (ablation) | D-PBEMO · Synthetic test problems | consultation workload | Hurt |
Surprising findings
- Authors report the optimal K does not significantly differ in performance, so K has little effect on results.
Limitations
- Regret analysis covers only the consultation module's optimal subset, not the exact solution of interest.
- Convergence of the whole D-PBEMO framework is not analyzed.
- Theorem on convergence of preference distribution is argued informally.
Experimental setup
Datasets: ZDT1-ZDT6 DTLZ1-DTLZ6 WFG1, WFG3, WFG5, WFG7 RNA inverse design tasks (Eterna) protein structure prediction (PSP) tasks
Benchmarks: ZDT DTLZ WFG RNA inverse design protein structure prediction
Metrics: approximation accuracy to golden point average accuracy to golden point RMSD
Baselines: I-MOEA/D-PLVF I-NSGA-II/LTR IEMO/D D-PBEMO-DTS D-PBEMO-PBO
Builds on: double Thompson sampling NSGA-II MOEA/D Copeland winner dueling bandits
Closest papers by meaning
- Neural Dueling Bandits: Preference-Based Optimization with Human Feedback (ICLR 2025 · similarity 0.89)
- Online Clustering of Dueling Bandits (ICML 2025 · similarity 0.87)
- Variance-aware Regret Bounds for Stochastic Contextual Dueling Bandits (ICLR 2024 · similarity 0.87)
- Preference Optimization on Pareto Sets: On a Theory of Multi-Objective Optimization (NeurIPS 2025 · similarity 0.86)
- Enhancing Preference-based Linear Bandits via Human Response Time (NeurIPS 2024 · similarity 0.86)
- Fusing Reward and Dueling Feedback in Stochastic Bandits (ICML 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.