Research map / Large language models / Alignment and preferences
MACPO: Weak-to-Strong Alignment via Multi-Agent Contrastive Preference Optimization
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes MACPO, a multi-agent iterative contrastive preference optimization framework where weak teachers (7-8B) and a strong student (70B) learn from each other's unfamiliar positive behaviors and penalize self-generated hard negatives, evaluated on HH-RLHF and PKU-SafeRLHF.
Problem
Weak-to-strong alignment: aligning strong student LLMs using noisy supervision from weaker teacher LLMs, where existing strong-to-weak and self-alignment methods are unsuitable and self-alignment tends to collapse over iterations.
Approach
Iteratively train weak teachers and a strong student on contrastive preference pairs: positive responses come from the other agents (unfamiliar, filtered by student perplexity), and negatives come from self-generated outputs of models fine-tuned on negative data (familiar hard negatives), optimized with DPO plus an SFT term.
Key results
- MACPO (iter3) reaches an average reward-model score of 67.85 vs 55.26 for the best strong-to-weak baseline (RLCD) and 56.57 for the best self-alignment baseline (Self-rewarding iter2).
- Under GPT-4o evaluation, MACPO beats Naive SFT on HH-Helpful with 89% win, and beats RLAIF on PKU-SafeRLHF with 49% win vs 16% lose.
- On MT-Bench, MACPO scores 4.63 vs 4.59 for RLCD and 4.16 for RLAIF; SPIN and Self-rewarding fall to 2.56 and 3.69.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use MACPO instead of RLCD | Llama2-70b weak-to-strong alignment · HH-Helpful | reward score: 52.77 -> 69.81 | Helped |
| use MACPO instead of Confident loss | Llama2-70b weak-to-strong alignment · HH-Harmless | reward score: 59.29 -> 70.25 | Helped |
| use MACPO instead of Self-rewarding | Llama2-70b weak-to-strong alignment · PKU-SafeRLHF | reward score: 60.38 -> 63.49 | Helped |
| use MACPO instead of RLCD | Llama2-70b alignment on general questions · MT-Bench | GPT-4 score: 4.59 -> 4.63 | Helped |
| add mutual positive behavior augmentation rather than use self-generated positive behavior (ablation) | MACPO strong student · HH-Helpful and HH-Harmless | alignment performance | Helped |
| add hard negative behavior construction rather than use negative behavior from weak teachers (ablation) | MACPO strong student · Helpfulness and harmlessness alignment datasets | alignment performance | Helped |
| iteratively train weak teachers instead of freezing them after initialization (ablation) | MACPO · Helpfulness and harmlessness alignment datasets | alignment performance | Helped |
| add perplexity filtering rather than randomly sample teacher labels (ablation) | MACPO strong student · HH-Helpful | reward score: 67.89 -> 69.81 | Helped |
| add perplexity filtering rather than randomly sample teacher labels (ablation) | MACPO strong student · HH-Harmless | reward score: 62.49 -> 70.25 | Helped |
| add perplexity filtering rather than randomly sample teacher labels (ablation) | MACPO strong student · PKU-SafeRLHF | reward score: 63.12 -> 63.49 | Helped |
| increase the number of weak teachers from one to three | MACPO · Weak-to-strong alignment | alignment performance | Helped |
Surprising findings
- SPIN's reward score drops after the first iteration (51.62 to 46.02 average), and Self-rewarding degrades after iteration 2, indicating collapse of self-alignment.
- Naive SFT and Confident loss score lower than the strong-to-weak baselines on the reward model average, despite being weak-to-strong methods.
Limitations
- Only evaluated on helpfulness and harmlessness alignment; not on math, code, or QA tasks.
- Negative behavior is induced only by fine-tuning on negative behavioral data; other jailbreak-style methods are not explored.
- Relies on a single third-party reward model and GPT-4o judge for automatic evaluation, and human evaluation uses only 100 samples per dataset.
Experimental setup
Datasets: HH-RLHF PKU-SafeRLHF MT-Bench
Benchmarks: HH-RLHF (Helpful, Harmless) test subsets PKU-SafeRLHF test set MT-Bench
Metrics: third-party reward model score GPT-4o pairwise win/tie/lose human pairwise win/tie/lose MT-Bench GPT-4 score
Baselines: RLAIF RLCD SPIN Self-rewarding Naive SFT Confident loss
Builds on: DPO SPIN Self-rewarding language models RLCD RLAIF Confident loss for weak-to-strong learning
Closest papers by meaning
- Weak-to-Strong Preference Optimization: Stealing Reward from Weak Aligned Model (ICLR 2025 · similarity 0.91)
- Super(ficial)-alignment: Strong Models May Deceive Weak Models in Weak-to-Strong Generalization (ICLR 2025 · similarity 0.90)
- Advantage-Guided Distillation for Preference Alignment in Small Language Models (ICLR 2025 · similarity 0.89)
- Bayesian WeakS-to-Strong from Text Classification to Generation (ICLR 2025 · similarity 0.88)
- HPS: Hard Preference Sampling for Human Preference Alignment (ICML 2025 · similarity 0.88)
- Weak-to-Strong Generalization: Eliciting Strong Capabilities With Weak Supervision (ICML 2024 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.