Research map / Large language models / Alignment and preferences

Improving Model Alignment Through Collective Intelligence of Open-Source Models

ICML 2025 · Large language models › Alignment and preferences · approach: Steering and correcting LLM behavior · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes Mixture of Agents Alignment (MoAA), which uses an ensemble of open-source LLMs as a data generator for SFT and as a pairwise reward model for DPO preference labeling, improving Llama-3.1-8B-Instruct and Gemma-2-9B-it.

Problem

Human-labeled alignment data is expensive and hard to scale, and synthetic data from a single strong (often proprietary) teacher has limited diversity and reproducibility. The paper asks whether collective open-source LLMs can supply better SFT and preference data.

Approach

Apply the Mixture-of-Agents (proposer/aggregator layered) architecture to generate SFT responses and to act as a pairwise reward model (with order-swapped evaluation and dynamically selected criteria) for labeling DPO preference pairs, so that the aligned model can match or exceed the MoA teacher and self-improve.

Key results

• Llama-3.1-8B-Instruct AlpacaEval 2 LC win rate rises from 22.33 to 57.23 and Arena-Hard from 19.5 to 48.3 after MoAA-SFT followed by MoAA-DPO.
• Gemma-2-9B-it-MoAA-DPO reaches 63.75 AlpacaEval 2 LC and 55.6 Arena-Hard, exceeding its base (47.43 / 42.0).
• Gemma-2-9B-it-MoAA-DPO achieves 90.6% of MoA performance at roughly 5.4% of its inference cost.

Measured effects reported in the paper

Surprising findings

• SFT on a small-scale MoA mixture let Gemma-2-9B-it outperform the strongest individual model in its own mixture, suggesting self-improvement beyond teachers.
• Naive combination of multi-model data (Combined 5, Random 5) underperformed single-model-generated synthetic data from GPT-4o in some settings, and MoA as reward model scored lower than ArmoRM on RewardBench yet produced a better DPO model on MT-Bench and Arena-Hard.

Limitations

• Architecture selection was done by manual/empirical search and an early BFGS attempt; no principled optimization of the mixture.
• Multi-turn synthetic data may suffer discontinuity; model-judge reward relies on GPT-4-based benchmarks that may be biased toward GPT-4 style.
• Benchmarks (AlpacaEval 2, Arena-Hard, MT-Bench) are judged by GPT-4 variants, and reasoning ability showed slight drops after MoAA-SFT for Llama.

Experimental setup

Datasets: UltraFeedback UltraChat-200k RewardBench PPE AlpacaEval 2 Arena-Hard MT-Bench MMLU HumanEval GPQA MATH

Benchmarks: AlpacaEval 2 Arena-Hard MT-Bench RewardBench PPE MMLU HumanEval GPQA MATH

Metrics: length-controlled win rate win rate MT-Bench score accuracy pass@1

Baselines: Combined 5 Random 5 Best of 5 (ArmoRM-Llama3-8B-v0.1) MAGPIE-Pro Meta-Rewarding LM GPT-4o-2024-05-13 data PairRM ArmoRM-Llama3-8B-v0.1 Llama-3.1-70B-Instruct as reward model Qwen2-72B-Instruct as reward model Gemma-2-27B-it as reward model

Builds on: Mixture of Agents DPO Magpie Meta-Rewarding LLM

Closest papers by meaning

• Mixture-of-Agents Enhances Large Language Model Capabilities (ICLR 2025 · similarity 0.91)
• Aligner: Efficient Alignment by Learning to Correct (NeurIPS 2024 · similarity 0.90)
• Collab: Controlled Decoding using Mixture of Agents for LLM Alignment (ICLR 2025 · similarity 0.89)
• Magpie: Alignment Data Synthesis from Scratch by Prompting Aligned LLMs with Nothing (ICLR 2025 · similarity 0.89)
• The Unlocking Spell on Base LLMs: Rethinking Alignment via In-Context Learning (ICLR 2024 · similarity 0.88)
• Learn Your Reference Model for Real Good Alignment (ICLR 2025 · similarity 0.88)