Research map / Large language models / Alignment and preferences
ComPO: Preference Alignment via Comparison Oracles
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes ComPO, a zeroth-order preference alignment method that uses pairwise comparison oracles (from likelihood changes on noisy preference pairs) to estimate sparse gradients of the output layer, applied after DPO on clean pairs, to reduce verbosity and likelihood displacement.
Problem
Direct alignment methods like DPO suffer from likelihood displacement and verbosity, which are driven by noisy preference pairs whose preferred and dispreferred responses have similar likelihood under the reference model.
Approach
Treat the preference data as a comparison oracle (sign of change in preferred/dispreferred likelihoods) and run a 1-bit, zeroth-order sparse gradient estimator with random perturbations of only the output layer weights, applied as a micro-finetuning step on noisy pairs after DPO on clean pairs.
Key results
- On Llama-3-Instruct-8B, DPO_clean+ComPO improves AlpacaEval 2 LC from 32.59 (DPO) to 35.79 and WR from 31.99 to 35.03 (best run).
- On Mistral-Instruct-7B, DPO_clean+ComPO reaches AlpacaEval 2 LC 26.17 vs 24.14 for DPO and MT-Bench avg 7.69 vs 5.86.
- SimPO+ComPO improves SimPO on Mistral-Instruct-7B (LC 40.22 to 42.27) and Gemma-2-it-9B (LC 60.36 to 62.42).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add ComPO on noisy pairs after DPO_clean | Mistral-Base-7B · AlpacaEval 2 | LC (%): 9.41 -> 11.66 | Helped |
| Add ComPO on noisy pairs after DPO_clean | Mistral-Instruct-7B · Arena-Hard | WR (%): 14.4 -> 10.5 | Hurt |
| Add ComPO on noisy pairs after DPO_clean | Llama-3-Base-8B · Arena-Hard | WR (%): 12.1 -> 12.1 | No clear effect |
| Add ComPO on noisy pairs after DPO_clean | Llama-3-Instruct-8B · AlpacaEval 2 | LC (%): 32.92 -> 35.79 | Helped |
| Add ComPO to SimPO | Mistral-Instruct-7B · AlpacaEval 2 | LC (%): 40.22 -> 42.27 | Helped |
| Increase perturbation count from m=800 to m=5400 (ablation) | ComPO on Mistral-Instruct-7B · AlpacaEval 2 | LC (%): 24.72 ± 1.02 (25.12) -> 26.49 ± 0.81 (27.20) | Helped |
| Perturb 3 layers instead of 1 (ablation) | ComPO on Mistral-7B-Instruct · AlpacaEval 2 | LC (%): 25.02 ± 0.91 (26.17) -> 26.00 ± 0.89 (27.09) | Helped |
| Set λg to retain 1%-5% of gradient entries rather than over-filtering or retaining all entries (ablation) | ComPO on Mistral-7B-Instruct · AlpacaEval 2 | WR (%): 15.72 ± 0.77 (16.34); 19.02 ± 0.62 (20.15); 19.21 ± 0.58 (20.25); 16.10 ± 0.11 (16.21) | Mixed |
| Increase noisy preference pairs from 100 to 300 (ablation) | ComPO on Mistral-7B-Instruct · AlpacaEval 2 | WR (%): 19.21 ± 0.58 (20.25) -> 20.07 ± 0.99 (21.35) | Helped |
| Increase noisy preference pairs from 100 to 300 (ablation) | ComPO on Mistral-7B-Instruct · Arena-Hard (GPT 4.1) | WR (%): 11.02 ± 0.13 (11.2) -> 11.76 ± 0.30 (12.1) | Helped |
Surprising findings
- DPO_clean does not always outperform DPO; log-likelihood margin filtering is not always helpful.
- DPO_clean+ComPO underperforms DPO on Arena-Hard for Mistral-Instruct-7B (14.4 vs 10.5 WR), which the authors attribute to shorter outputs and judge length bias.
Limitations
- Only the output layer (or a few layers) is perturbed; full-parameter perturbation was not tested due to compute limits.
- Purely offline method limited to the preference dataset; the noise threshold delta=3 strongly affects results.
- Theoretical convergence relies on an assumed smooth nonconvex function compatible with comparison oracles; best-iterate guarantee is not practically computable.
Experimental setup
Datasets: UltraFeedback (trl-lib/ultrafeedback_binarized) HuggingFaceH4/ultrafeedback_binarized
Benchmarks: AlpacaEval 2 Arena-Hard MT-Bench
Metrics: length-controlled win rate win rate MT-Bench score log-likelihood of preferred and dispreferred responses
Baselines: DPO DPO_clean SimPO
Builds on: DPO SimPO 1-bit compressed sensing zeroth-order optimization
Closest papers by meaning
- Cal-DPO: Calibrated Direct Preference Optimization for Language Model Alignment (NeurIPS 2024 · similarity 0.94)
- TODO: Enhancing LLM Alignment with Ternary Preferences (ICLR 2025 · similarity 0.94)
- SimPER: A Minimalist Approach to Preference Alignment without Hyperparameters (ICLR 2025 · similarity 0.93)
- Less is More: Improving LLM Alignment via Preference Data Selection (NeurIPS 2025 · similarity 0.93)
- Robust LLM Alignment via Distributionally Robust Direct Preference Optimization (NeurIPS 2025 · similarity 0.93)
- Noise Contrastive Alignment of Language Models with Explicit Rewards (NeurIPS 2024 · similarity 0.93)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.