Research map / Large language models / Alignment and preferences
Aligning Compound AI Systems via System-level DPO
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes SysDPO, a DPO extension that models compound AI systems as DAGs and aligns all components jointly end-to-end, with SysDPO-Direct (observed intermediate outputs) and SysDPO-Sampling (sampled intermediates via diverse beam search).
Problem
Aligning compound AI systems (multiple interacting LLMs, diffusion models, tools) with human preferences is hard because interactions are non-differentiable and system-level preferences cannot be decomposed into component-level preferences.
Approach
Factorize the system likelihood along a DAG of component generations so the DPO loss decomposes into per-component terms; SysDPO-Direct uses observed intermediate outputs in a system-specific preference dataset, while SysDPO-Sampling approximates the marginal over intermediates with a small set of diverse high-probability samples from diverse beam search, enabling end-to-end gradient optimization.
Key results
- On the LLM+diffusion task, SysDPO-Direct reaches Preference Score 0.25 and 73% Order Consistency Ratio, versus 0.23 / 65% for training only the language model and -0.20 / 32% for the system before alignment.
- On the two-Qwen1.5-1.8B collaboration system, SysDPO-Sampling reaches 19.8% WR-chosen (vs 12.8% prompted system, 16.6% Separate-DPO) and 66.4% WR-prompted.
- In an ablation, DBS with 2 samples gets 68.5% WR-prompted, while Monte Carlo sampling with 2 to 5 samples gets 66.0 to 67.0%.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| jointly align the LLM and diffusion model with SysDPO-Direct | Llama-3-8B + Stable Diffusion 1.5 · LLM + diffusion image sequences | Preference Score: -0.20 -> 0.25 | Helped |
| jointly align the LLM and diffusion model with SysDPO-Direct | Llama-3-8B + Stable Diffusion 1.5 · LLM + diffusion image sequences | OC Ratio: 32% -> 73% | Helped |
| train only the language model and freeze the diffusion model (ablation) | Llama-3-8B + Stable Diffusion 1.5 · LLM + diffusion image sequences | OC Ratio: 32% -> 65% | Helped |
| train only the diffusion model and freeze the language model (ablation) | Llama-3-8B + Stable Diffusion 1.5 · LLM + diffusion image sequences | OC Ratio: 32% -> 38% | Helped |
| jointly align both LLM stages instead of using prompting alone | Qwen1.5-1.8B-Chat two-stage system · Two-LLM collaboration | WR-Chosen: 12.8% to 19.8% | Helped |
| jointly align both LLM stages instead of separate DPO | Qwen1.5-1.8B-Chat two-stage system · Two-LLM collaboration | WR-Prompted: 57.3 -> 66.4 | Helped |
| train ψ1 and freeze ψ2 (ablation) | SysDPO-Sampling two-stage system · Two-LLM collaboration | WR-Chosen: 12.8 -> 16.0 | Helped |
| train ψ2 and freeze ψ1 (ablation) | SysDPO-Sampling two-stage system · Two-LLM collaboration | WR-Chosen: 12.8 -> 18.1 | Helped |
| train ψ1 and freeze ψ2 rather than jointly training both stages (ablation) | SysDPO-Sampling two-stage system · Two-LLM collaboration | WR-Chosen: 19.8 -> 16.0 | Hurt |
| train ψ2 and freeze ψ1 rather than jointly training both stages (ablation) | SysDPO-Sampling two-stage system · Two-LLM collaboration | WR-Chosen: 19.8 -> 18.1 | Hurt |
| use Diverse Beam Search rather than Monte Carlo sampling with 2 candidates (ablation) | SysDPO-Sampling · Two-LLM collaboration | WR-Prompted: 66.8 -> 68.5 | Helped |
| increase DBS sample size from 2 to 4 (ablation) | SysDPO-Sampling · Two-LLM collaboration | WR-Prompted: 68.5 -> 68.2 | No clear effect |
Surprising findings
- Training only the language model outperforms training only the diffusion model by a wide margin, and training only the diffusion model yields little gain over the unaligned system.
- Two intermediate samples via diverse beam search appear sufficient, with larger sample counts not improving results.
Limitations
- SysDPO-Direct needs observed intermediate outputs, which may be costly for high-dimensional or latent components.
- Experiments are limited to small systems (two components, one three-LLM extension with only preliminary results); the three-LLM result is reported without a full table.
- Evaluation for the LLM+diffusion task relies on a hand-designed preference score and a synthetic dataset with GPT-4 generated prompts.
Experimental setup
Datasets: Intel/orca-dpo-pairs Synthetic LLM+diffusion preference dataset (6,000 comparisons)
Benchmarks: Intel/orca-dpo-pairs held-out split (193 instructions) Synthetic image-sequence progression benchmark
Metrics: Preference Score Order Consistency Ratio win rate (WR-chosen) win rate (WR-prompted)
Baselines: System Before Alignment Best-of-N sampling Only Train Language Model Only Train Diffusion Model Prompted System Separate-DPO
Builds on: DPO Bradley-Terry model Diverse Beam Search DDPM
Closest papers by meaning
- 3D-Properties: Identifying Challenges in DPO and Charting a Path Forward (ICLR 2025 · similarity 0.87)
- Robust Multi-Objective Preference Alignment with Online DPO (AAAI 2025 · similarity 0.87)
- AlphaPO: Reward Shape Matters for LLM Alignment (ICML 2025 · similarity 0.87)
- Beyond Reverse KL: Generalizing Direct Preference Optimization with Diverse Divergence Constraints (ICLR 2024 · similarity 0.86)
- Linear Alignment: A Closed-form Solution for Aligning Human Preferences without Tuning and Feedback (ICML 2024 · similarity 0.86)
- Understanding the Logic of Direct Preference Alignment through Logic (ICML 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.