Research map / Image generation / Text-to-image diffusion
Rewarded soups: towards Pareto-optimal alignment by interpolating weights fine-tuned on diverse rewards
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's title only (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Interpolate weights fine-tuned on two diverse rewards instead of using separate reward-specialized models | LLaMA RLHF summarization · LLaMA RLHF summarization (first reported comparison; dataset unspecified) | hypervolume: 0.367 vs 0.340 | Hurt |
| Interpolate weights fine-tuned on two diverse rewards instead of using separate reward-specialized models | LLaMA RLHF summarization · LLaMA RLHF summarization (second reported comparison; dataset unspecified) | hypervolume: 1.176 vs 1.186 | Helped |
| Interpolate reward-specialized weights rather than train MORL scalarizations | COCO image captioning · COCO | hypervolume: 0.140 vs 0.140 | No clear effect |
| Uniformly average five weights, each fine-tuned on a different reward (ablation) | ExpansionNetv2 image captioning model · COCO | captioning rewards (BLEU1, BLEU4, ROUGE, METEOR, CIDEr) | Mixed |
| Add an additional reward-specialized network to the weight average (ablation) | Rewarded soup for image captioning · COCO | coverage across captioning rewards | Helped |
| Interpolate the AVA- and CAFE-specialized weights | Text-to-image diffusion model · Text-to-image generation | AVA and CAFE reward scores | Helped |
| Fine-tune on the AVA reward instead of using the CAFE-specialized model | Text-to-image diffusion model · Text-to-image generation | CAFE reward | Helped |
| Fine-tune on small-object accuracy rather than large-object accuracy | UnIVAL visual grounding model · RefCOCO+ | large-object accuracy | Hurt |
| Uniformly average weights specialized for different object sizes (ablation) | UnIVAL visual grounding model · RefCOCO+ | accuracy across object sizes | Helped |
| Interpolate weights specialized for risky and cautious locomotion rewards | Humanoid locomotion policy · Brax locomotion | trade-off between risk-taking and cautiousness | Mixed |
| Use linear weight interpolation between reward-specialized models | Text-to-text, image captioning, image generation, visual grounding, VQA, and locomotion models · Multiple tasks and reward pairs | reward trade-off | Helped |
Closest papers by meaning
- Rewards-in-Context: Multi-objective Alignment of Foundation Models with Dynamic Preference Adjustment (ICML 2024 · similarity 0.90)
- TMPO: Trajectory Matching Policy Optimization for Diverse and Efficient Diffusion Alignment (NeurIPS 2026 · similarity 0.89)
- Direct Alignment with Heterogeneous Preferences (NeurIPS 2025 · similarity 0.89)
- Scaling Laws for Reward Model Overoptimization in Direct Alignment Algorithms (NeurIPS 2024 · similarity 0.89)
- MPO: An Efficient Post-Processing Framework for Mixing Diverse Preference Alignment (ICML 2025 · similarity 0.89)
- MaxMin-RLHF: Alignment with Diverse Human Preferences (ICML 2024 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.