Research map / Large language models / Alignment and preferences

Distributionally Robust Token Optimization in RLHF

NeurIPS 2026 · Large language models › Alignment and preferences · approach: Robust and calibrated DPO · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes DRTO, which applies span-level distributionally robust optimization (KL and chi-square ambiguity sets) to the actor loss of Reinforced Token Optimization (RTO) for more robust math and code reasoning under distribution shift.

Problem

LLM reasoning accuracy is brittle under small distribution shifts in wording, format, or language, and existing robust PPO/DPO variants are conservative or costly and under-use token-level credit assignment.

Approach

Partition each response into low-confidence-cut spans, compute mean PPO loss per span, and apply f-divergence DRO over spans in the RTO actor update: KL gives an exponential-tilt (entropic) reweighting, chi-square gives a mean-plus-std surrogate.

Key results

• chi2-DRTO reaches 22.4 on MATH-500 vs 18.0 for RTO (+4.4 points) on Llama3-8B.
• KL-DRTO gains +2.7 points on LiveCodeBench over RTO on Qwen3-8B, with +9.6 on MBPP.
• DRTO adds essentially no runtime overhead over RTO, while FlowRL and DAPO are several times more expensive.

Surprising findings

• DPO falls well below its SFT initialization on HumanEval (-6.71) and LiveCodeBench (-16.41), attributed to offline over-optimization on coding data.
• SFT outperforms RTO on MBPP and LiveCodeBench, so the RTO baseline is not uniformly strong on the code pipeline.

Limitations

• Evaluated only on two 8B backbones and single-turn settings.
• Each configuration reported from a single training seed.
• No formal guarantees against adversarial or worst-case safety failures.

Experimental setup

Datasets: OpenMathInstruct UltraInteract UltraFeedback GSM8K

Benchmarks: GSM8K GSM (CoT) GSM (ES) MATH-500 GPQA HumanEval MBPP LiveCodeBench

Metrics: accuracy pass@1

Baselines: RTO SFT DPO PPO FlowRL DAPO DRPPO trajectory-level DRTO

Builds on: Reinforced Token Optimization PPO DPO Distributionally Robust Optimization GAE

Closest papers by meaning

• STAPO: Stabilizing Reinforcement Learning for LLMs by Silencing Rare Spurious Tokens (NeurIPS 2026 · similarity 0.89)
• Do Not Let Low-Probability Tokens Over-Dominate in RL for LLMs (ICLR 2026 · similarity 0.89)
• DPO Meets PPO: Reinforced Token Optimization for RLHF (ICML 2025 · similarity 0.89)
• Robust Reinforcement Learning from Corrupted Human Feedback (NeurIPS 2024 · similarity 0.88)
• MA-RLHF: Reinforcement Learning from Human Feedback with Macro Actions (ICLR 2025 · similarity 0.88)
• Diversity-Aware Policy Optimization for Large Language Model Reasoning (NeurIPS 2025 · similarity 0.87)