Research map / Large language models / Alignment and preferences

Aligning Language Model Benchmarks with Pairwise Preferences

NeurIPS 2026 · Large language models › Alignment and preferences · approach: Steering and correcting LLM behavior · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Introduces benchmark alignment, reweighting static benchmark items via learning-to-rank (RankNet-style BenchAlign) on pairwise reward-model preferences so benchmarks rank unseen, larger models per helpfulness/honesty preferences.

Problem

Static language model benchmarks often fail to predict downstream utility and rank models differently from pairwise user preferences, and there is no systematic way to update benchmarks to align with such preferences.

Approach

Learn continuous per-item weights over existing benchmark questions with a pairwise learning-to-rank loss so that the weighted score reproduces reward-model-derived pairwise model rankings, then test generalization to unseen (larger) models.

Key results

• On 70B+ target models with ArmoRM (Helpful) preferences, BenchAlign reaches Acc_pair 0.778 and Spearman rho 0.707, versus 0.542 and 0.154 for No Alignment.
• With 13B+ targets and ArmoRM (Honest), BenchAlign achieves rho 0.686 versus 0.251 for No Alignment.
• An oracle selection of about 20 well-chosen neighboring models can reach alignment comparable to using all models, and TinyBenchmarks distillation to ~1,500 questions with BenchAlign matches or slightly exceeds full alignment.

Surprising findings

• Benchmarks trained on small-to-medium models generalize to substantially larger (70B+) unseen models in most settings.
• Learned weights draw signal from all benchmarks, including unrelated Instruction Following, rather than topically related tasks (e.g., Geometry for physics target).

Limitations

• Relies on reward models as proxies for real-world human preferences, which may not capture preference heterogeneity.
• Requires ground-truth preference rankings; alignment is only useful when existing benchmarks already correlate positively with target preferences.
• Naive alignment is data-intensive, needing around 1,000 models and 5,000 questions.

Experimental setup

Datasets: OpenLLMLeaderboard HelpSteer UltraFeedback Big Bench Hard MMLU Pro MuSR MATH GPQA IFEval

Benchmarks: OpenLLMLeaderboard (BBH, MMLU-Pro, MuSR, MATH, GPQA, IFEval)

Metrics: Spearman rank correlation pairwise ranking accuracy

Baselines: No Alignment Random Individual MetaBench TinyBenchmarks

Builds on: RankNet LambdaRank TinyBenchmarks MetaBench Item Response Theory

Closest papers by meaning

• Aligner: Efficient Alignment by Learning to Correct (NeurIPS 2024 · similarity 0.91)
• Clean First, Align Later: Benchmarking Preference Data Cleaning for Reliable LLM Alignment (NeurIPS 2025 · similarity 0.90)
• BenchmarkCards: Standardized Documentation for Large Language Model Benchmarks (NeurIPS 2025 · similarity 0.90)
• Principled Data Selection for Alignment: The Hidden Risks of Difficult Examples (ICML 2025 · similarity 0.90)
• Theoretical Limits of Language Model Alignment (NeurIPS 2026 · similarity 0.90)
• Restoring Calibration for Aligned Large Language Models: A Calibration-Aware Fine-Tuning Approach (ICML 2025 · similarity 0.89)