Research map / Large language models / Alignment and preferences

Oracle-Robust Online Alignment for Large Language Models

NeurIPS 2026 · Large language models › Alignment and preferences · approach: Robust and calibrated DPO · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes an oracle-robust version of SAIL online LLM alignment with a pointwise preference-probability uncertainty set, shows the robust loss decomposes into SAIL loss plus a sensitivity penalty for log-linear policies, and proves O~(eps^-2) oracle complexity to Moreau-envelope stationarity via projected stochastic composite gradient.

Problem

Online LLM alignment with preference feedback from a misspecified oracle that deviates from an unknown true oracle; on-policy feedback loops can amplify oracle quirks, and existing online methods assume the oracle is correct while robust DPO methods target static datasets.

Approach

Define a pointwise oracle uncertainty set on preference probabilities, form the worst-case SAIL loss, and show under log-linear policies and BT structure that it equals SAIL loss plus rho*beta times an expected absolute log-ratio (pairwise score) penalty, which is weakly convex and admits projected stochastic composite updates.

Key results

• Shows L^W_rho(theta) = L_SAIL(theta) + rho*beta*R(theta) exactly for log-linear policies under the pointwise uncertainty set.
• Proves the penalty R(theta) is kappa_R-weakly convex with kappa_R <= 16B^2 + 4DB^3, and the composite objective is weakly convex with kappa = L_SAIL + lambda*kappa_R.
• Proves O~(eps^-2) sample complexity (BT = O~(eps^-2)) for reaching eps-stationarity of the Moreau envelope.

Surprising findings

• The penalty R(theta) is not convex in general, shown by a one-prompt two-response counterexample, so weak convexity is needed instead.

Limitations

• No empirical experiments validating the robust objective or the algorithm on LLMs.
• Requires log-linear policy class and bounded feasible set and features, which do not hold for neural LLM policies as used in practice.
• Convergence is to stationarity of a surrogate (Moreau envelope), not to global optimality or robustness guarantees on preference shift.

Experimental setup

Builds on: SAIL DPO Bradley-Terry model Moreau envelope

Closest papers by meaning

• ComPO: Preference Alignment via Comparison Oracles (NeurIPS 2025 · similarity 0.91)
• Alignment of Large Language Models with Constrained Learning (NeurIPS 2025 · similarity 0.90)
• Robust LLM Alignment via Distributionally Robust Direct Preference Optimization (NeurIPS 2025 · similarity 0.89)
• Noise Contrastive Alignment of Language Models with Explicit Rewards (NeurIPS 2024 · similarity 0.89)
• Doubly Robust Alignment for Large Language Models (NeurIPS 2025 · similarity 0.89)
• Distributionally Robust Listwise Preference Optimization (NeurIPS 2026 · similarity 0.89)