Research map / Reinforcement learning / RL from feedback
COMAL: A Convergent Meta-Algorithm for Aligning LLMs with General Preferences
Yixin Liu, Argyris Oikonomou, Weiqiang Zheng, Yang Cai, Arman Cohan
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| update the reference policy every 6 iterations (COMAL) instead of fixed reference (ablation) | INPO (Llama-3-8B-Instruct) · preference-oracle win rate | avg win rate (%): 63.44 -> 71.37 | Improved |
| update the reference policy every 6 iterations (COMAL) instead of fixed reference (ablation) | INPO (Qwen2.5-7B-SFT) · preference-oracle win rate | avg win rate (%): 64.19 -> 68.93 | Improved |
| COMAL vs per-iteration reference update (Iter-IPO) (ablation) | COMAL · preference-oracle head-to-head, Llama-3 | win rate (%): 60.25 vs Iter-IPO | Improved |
| use small KL regularization over many iterations (Iter-IPO, INPO) | Iter-IPO/INPO on Llama-3 · training stability | performance | Hurt |
| use larger KL regularization (Iter-IPO-L) | Iter-IPO · Llama-3 preference-oracle win rate | win rate (%): Iter-IPO beats Iter-IPO-L 50.68 | Mixed |
| iterative methods vs one-shot DPO/IPO | Iter-IPO vs IPO · preference-oracle win rate Llama-3 | win rate (%): 46.16 -> 66.93 avg | Improved |
| preference optimization (IPO) vs BASE | IPO · HumanEval Llama-3 | pass rate: 79.2 -> 65.3 | Hurt |
| COMAL vs BASE on reasoning | COMAL · GSM8K Llama-3 | accuracy: 77.0 -> 77.5 | No clear effect |
| Iter-IPO vs BASE | Iter-IPO · GSM8K Llama-3 | accuracy: 77.0 -> 71.5 | Hurt |
| COMAL vs Iter-IPO on Arena-Hard | COMAL · Arena-Hard Llama-3 | score: 43.8 -> 41.3 | Hurt |
| preference optimization vs SFT base on GSM8K | IPO/DPO/INPO/COMAL · GSM8K Qwen2.5-7B-SFT | accuracy: 77.5 -> 91.0 | Improved |
| COMAL vs BASE on AlpacaEval | COMAL · AlpacaEval Llama-3 | win rate (%): 25.0 -> 53.5 | Improved |
Closest papers by meaning
- Self-Play Preference Optimization for Language Model Alignment (ICLR 2025 · similarity 0.94)
- Iterative Nash Policy Optimization: Aligning LLMs with General Preferences via No-Regret Learning (ICLR 2025 · similarity 0.94)
- Magnetic Preference Optimization: Achieving Last-iterate Convergence for Language Model Alignment (ICLR 2025 · similarity 0.92)
- Nash Learning from Human Feedback (ICML 2024 · similarity 0.91)
- MaxMin-RLHF: Alignment with Diverse Human Preferences (ICML 2024 · similarity 0.91)
- Online Preference Alignment for Language Models via Count-based Exploration (ICLR 2025 · similarity 0.91)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
