Research map / Large language models / Alignment and preferences
Sparta Alignment: Collectively Aligning Multiple Language Models through Combat
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Sparta Alignment, where a pool of Qwen2.5-7B models compete in reputation-weighted peer-judged duels and are aligned with DPO on the resulting preference pairs.
Problem
Single-model self-alignment is unreliable because a model is biased when judging its own outputs and produces homogeneous responses, limiting preference learning.
Approach
Multiple LLMs form a tribe that duels on instructions while other members judge the responses; judgments are aggregated with Elo-style reputation weights, and winners/losers gain/lose reputation, with all models then trained by DPO on the peer-generated preference pairs.
Key results
- Sparta achieves best results on 10 of 12 dataset settings, e.g., Alpaca 7.12 vs 5.36 for best initial model and 5.56 for the strongest baseline.
- On GSM8K Sparta reaches 0.813 vs 0.778 for the best initial model (about 4.5% gain).
- Averaged across tasks, Sparta reports a 7.0% average improvement over initial models and baselines.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use collective Sparta alignment instead of the best initial model | Qwen2.5-7B-Instruct pool · GSM8K | pass@1 accuracy: 4.5% | Helped |
| use collective Sparta alignment instead of the strongest self-alignment baseline | Qwen2.5-7B-Instruct pool · COM² | LLM judge score: 9.6% | Helped |
| use Sparta instead of the best initial model | Qwen2.5-7B-Instruct pool · Alpaca | LLM judge score: 32.8% | Helped |
| add reputation updates (ablation) | Sparta · Alpaca | LLM judge score: 6.17 -> 7.12 | Helped |
| add judge downweighting (ablation) | Sparta · Alpaca | LLM judge score: 6.63 -> 7.12 | Helped |
| include randomness in matchmaking (ablation) | Sparta · Alpaca | LLM judge score: 6.04 -> 7.12 | Helped |
| constrain matchmaking to top-k similar-reputation models (ablation) | Sparta · Alpaca | LLM judge score: 6.74 -> 7.12 | Helped |
| remove the top-k matchmaking constraint (ablation) | Sparta · Alpaca | LLM judge score: 6.74 -> 7.12 | Hurt |
| increase model pool size from 3 to 10 | Sparta · Alpaca | LLM judge score: 19.1% | Helped |
| increase model pool size from 3 to 10 | Sparta · COM² | LLM judge score: 33.1% | Helped |
| increase diversity of the initial model pool to 10 distinct models rather than 1 repeated model | Sparta · Alpaca, COM², and MedQA | performance: 18.5% averaged across the three datasets | Helped |
| collectively align 10 models rather than one model | instruction-following models · Instruction-following response diversity | Edit: 0.732 -> 0.722 | Mixed |
Surprising findings
- A model that initially underperforms can become the strongest in the pool through collective alignment.
- Output diversity decreases over iterations even as test performance improves.
Limitations
- Model diversity and performance gains diminish over iterations, so improvements may not be unbounded.
- Requires more computational resources than single-model methods.
- Limited gains on KCross, attributed to the small dataset size; potential bias propagation from biased pool members.
Experimental setup
Datasets: Tulu-v2 Alpaca MedQA-US Normad Knowledge Crosswords COM2 GSM8K MATH TruthfulQA
Benchmarks: MedQA Normad Knowledge Crosswords GSM8K COM2 MATH Alpaca TruthfulQA
Metrics: pass@1 accuracy LLM-as-a-judge score (0-10) log-probability truthfulness edit distance / BLEU / embedding diversity
Baselines: Best Init Self-Rewarding Meta-Rewarding SPIN SPPO
Builds on: DPO Self-Rewarding Meta-Rewarding SPIN SPPO
Closest papers by meaning
- On Evaluating LLM Alignment by Evaluating LLMs as Judges (NeurIPS 2025 · similarity 0.88)
- RMB: Comprehensively benchmarking reward models in LLM alignment (ICLR 2025 · similarity 0.88)
- Your Weak LLM is Secretly a Strong Teacher for Alignment (ICLR 2025 · similarity 0.87)
- Align-Pro: A Principled Approach to Prompt Optimization for LLM Alignment (AAAI 2025 · similarity 0.87)
- LLM Alignment as Retriever Optimization: An Information Retrieval Perspective (ICML 2025 · similarity 0.87)
- Stackelberg Self-Annotation: A Robust Approach to Data-Efficient LLM Alignment (NeurIPS 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.