Research map / Bandits and online learning / Contextual bandits and Thompson sampling

Neural Dueling Bandits: Preference-Based Optimization with Human Feedback

ICLR 2025 · Bandits and online learning › Contextual bandits and Thompson sampling · approach: Practical contextual bandit extensions · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes neural dueling bandit algorithms (NDB-UCB and NDB-TS) that estimate a non-linear reward with a neural network trained on binary preference feedback via cross-entropy, with sub-linear regret guarantees, and extends to neural contextual bandits with binary feedback.

Problem

Contextual dueling bandits assume a linear reward, which fails for non-linear real-world preference structures; the goal is to find the best arm per context from noisy pairwise preference feedback when the reward is non-linear.

Approach

Estimate the non-linear latent reward with a fully connected ReLU network trained by cross-entropy (BTL likelihood) on pairwise preferences, then select duels via UCB or Thompson sampling using NTK-based confidence bounds on reward differences.

Key results

• NDB-UCB and NDB-TS outperform linear baselines LinDB-UCB, LinDB-TS and CoLSTIM on Square and Cosine reward functions, since linear methods incur linear regret on non-linear rewards.
• Theoretical regret bounds of order Õ((sqrt(d~)/κ_μ + B sqrt(λ/κ_μ)) sqrt(T d~)) are sub-linear when effective dimension d~ = Õ(sqrt(T)).

Measured effects reported in the paper

Surprising findings

• The effective dimension of the dueling setting is larger than in standard neural bandits, making the regret bounds worse than NeuralUCB's bound.

Limitations

• Regret bounds are weaker than standard neural bandits because the effective dimension scales with the number of pairwise contexts.
• Sub-linear regret requires the effective dimension assumption d~ = Õ(sqrt(T)).
• Experiments are limited to synthetic reward functions with no hyperparameter tuning for the neural methods.

Experimental setup

Datasets: synthetic reward functions (Square, Cosine) with uniform random features

Benchmarks: synthetic Square reward function synthetic Cosine reward function

Metrics: average cumulative regret weak cumulative regret

Baselines: LinDB-UCB LinDB-TS CoLSTIM

Builds on: NeuralUCB NeuralTS GP-UCB linear contextual dueling bandits

Closest papers by meaning

• Variance-aware Regret Bounds for Stochastic Contextual Dueling Bandits (ICLR 2024 · similarity 0.93)
• Nearly Optimal Algorithms for Contextual Dueling Bandits from Adversarial Feedback (ICML 2025 · similarity 0.92)
• Feel-Good Thompson Sampling for Contextual Dueling Bandits (ICML 2024 · similarity 0.92)
• Borda Regret Minimization for Generalized Linear Dueling Bandits (ICML 2024 · similarity 0.91)
• Bandits with Ranking Feedback (NeurIPS 2024 · similarity 0.90)
• Online Clustering of Dueling Bandits (ICML 2025 · similarity 0.89)