Research map / Large language models / Alignment and preferences

Stackelberg Self-Annotation: A Robust Approach to Data-Efficient LLM Alignment

NeurIPS 2025 · Large language models › Alignment and preferences · approach: Robust and calibrated DPO · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes SGPO, a Stackelberg game between a policy and a worst-case Wasserstein-ball preference distribution, and SSAPO, a self-annotation DPO-style instantiation that uses 2K seed UltraFeedback preferences plus iterative self-labeling and DRO reweighting.

Problem

Preference alignment of LLMs typically needs large curated preference data that is costly and noisy; self-annotated synthetic labels can be biased and compound over iterations.

Approach

Models alignment as a Stackelberg game in preference-gap space where the follower picks a worst-case distribution within a W1 ball around the empirical gap distribution; approximates the concave log-sigmoid inner objective with a max-of-affine surrogate to obtain a finite convex program whose solution reweights self-annotated pairs.

Key results

• SSAPO on LLaMA3-8B reaches 33.33% length-controlled win rate and 40.12% raw win rate vs GPT-4 on AlpacaEval 2.0 using 2K seed preferences, above SPA's 21.85%/24.95%.
• Mistral-7B-SSAPO achieves 24.44% LC win rate and 35.82% win rate vs GPT-4, versus 9.03%/7.68% for Mistral-7B-DPO.
• Under 25% seed label flips, Mistral-SSAPO drops from 26.90% to 19.70% LC win rate, while LLaMA-SSAPO rises from 33.33% to 43.74%.

Measured effects reported in the paper

Surprising findings

• LLaMA-SSAPO improves under 25% seed label corruption (33.33% to 43.74% LC win rate), which the authors attribute to implicit regularization.
• Larger piecewise-linear tangent count K=7 hurts performance (19.05% vs 23.20% at K=6), which the authors attribute to solver instability.

Limitations

• Scalability limited by number of preferences N due to convex program size; relies on uniform grouping heuristic that is not exact.
• Guarantees target training-time robustness to preference noise and mild shifts, not inference-time adversarial prompts or jailbreaks.
• Theoretical results rely on assumptions such as compact parameter set and strong concavity for local convergence; the paper's own proofs contain gaps (e.g., regret bounds and the claimed ε-dependence).

Experimental setup

Datasets: UltraFeedback UltraChat

Benchmarks: AlpacaEval 2.0 MT-Bench

Metrics: length-controlled win rate win rate vs GPT-4 average MT-Bench score

Baselines: DPO Iter DPO (PairRM) Iter DPO (LLM-as-judge) SPA Zephyr-7B-beta Mistral-Large

Builds on: DPO SPA Iter DPO Stackelberg Alignment RLHF

Closest papers by meaning

• Spread Preference Annotation: Direct Preference Judgment for Efficient LLM Alignment (ICLR 2025 · similarity 0.92)
• Leveraging robust optimization for llm alignment under distribution shifts (NeurIPS 2025 · similarity 0.92)
• Panacea: Pareto Alignment via Preference Adaptation for LLMs (NeurIPS 2024 · similarity 0.91)
• TODO: Enhancing LLM Alignment with Ternary Preferences (ICLR 2025 · similarity 0.91)
• Self-Play Preference Optimization for Language Model Alignment (ICLR 2025 · similarity 0.90)
• Robust LLM Alignment via Distributionally Robust Direct Preference Optimization (NeurIPS 2025 · similarity 0.90)