Research map / Reinforcement learning / RL from feedback

Iterative Foundation Model Fine-Tuning on Multiple Rewards

NeurIPS 2025 · Reinforcement learning › RL from feedback · approach: RL fine-tuning of generative models · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes IterativeRS, which fine-tunes foundation models with PPO on each reward separately and periodically merges expert policies (generalizing MORLHF and Rewarded Soups), with a convergence analysis for convex losses and experiments on molecules, DNA, and summarization.

Problem

Fine-tuning foundation models on multiple, possibly conflicting reward objectives, where single-reward scalarization yields high variance across objectives and naive expert merging can degrade performance.

Approach

Fine-tune one expert per objective but periodically (every m steps) merge experts into a shared policy that restarts the next round of objective-specific fine-tuning, optionally updating only a random subset of M objectives per round; this interpolates between reward combining (m=1) and Rewarded Soups (m=T).

Key results

• On small molecule generation, IterativeRS reaches Avg Reward 1.4017 versus 1.3062 for Rewarded Soups, 1.2910 for MORLHF, and 0.6765 for RiC (but ICV 3.5854 is lower than RS's 4.2674).
• On DNA sequence generation, IterativeRS achieves Avg Reward 0.5927 (about 1% above RiC's 0.5842) with ICV 3.8310 versus RiC's 2.4672.
• On Reddit summarization, IterativeRS obtains Avg Score 0.4849 and ICV 4.9134, the best among MORLHF (0.4525), RS (0.4732), and RiC (0.4518).

Measured effects reported in the paper

Surprising findings

• On molecule generation, IterativeRS has lower ICV (3.5854) than Rewarded Soups (4.2674), contrary to the claim of more balanced performance.
• Selective merging (choosing among ten merge weight sets by reward) gave no improvement over fixed uniform merging in the RLOO DNA experiment.

Limitations

• Convergence theory assumes convex, L-smooth, strongly convex losses, which the authors acknowledge may not hold for LLM policies.
• Tighter theoretical bound does not guarantee better deployment performance; the effect of m depends on the loss and dataset.
• Merge weight selection in text summarization relies on a set of seven candidate weight vectors chosen by reward on validation samples.

Experimental setup

Datasets: MOSES QM9 MPRA Reddit Summary dataset

Benchmarks: QM9 (PAMNet-evaluated molecules) MPRA (Malinois-evaluated DNA sequences) Reddit post summarization

Metrics: average reward per objective average reward across objectives ICV (inverse coefficient of variation) ROUGE Avg Score

Baselines: MORLHF Rewarded Soups Rewards-in-Context

Builds on: Rewarded Soups PPO MORLHF Rewards-in-Context

Closest papers by meaning

• Towards Cost-Effective Reward Guided Text Generation (ICML 2025 · similarity 0.85)
• Bridging Model-Based Optimization and Generative Modeling via Conservative Fine-Tuning of Diffusion Models (NeurIPS 2024 · similarity 0.85)
• Feedback Efficient Online Fine-Tuning of Diffusion Models (ICML 2024 · similarity 0.85)
• Score as Action: Fine Tuning Diffusion Generative Models by Continuous-time Reinforcement Learning (ICML 2025 · similarity 0.85)
• GRAM: A Generative Foundation Reward Model for Reward Generalization (ICML 2025 · similarity 0.85)
• Reward Fine-Tuning Two-Step Diffusion Models via Learning Differentiable Latent-Space Surrogate Reward (CVPR 2025 · similarity 0.85)