Research map / Large language models / Alignment and preferences

Reward-Guided Prompt Evolving in Reinforcement Learning for LLMs

ICML 2025 · Large language models › Alignment and preferences · approach: Reward modeling for diverse preferences

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes eva, an asymmetric self-play method where a prompt-creator policy estimates prompt informativeness via reward advantage and evolves high-regret prompts, improving gemma-2-9b-it across DPO, RLOO and other RLHF algorithms.

Problem

Standard RLHF for LLMs trains on a static, pre-curated prompt set sampled on a fixed schedule, regardless of each prompt's usefulness to the current policy, which limits training efficiency and generalization.

Approach

Treat RL post-training as an asymmetric minimax-regret game: a creator estimates prompt informativeness by reward advantage (best minus baseline response reward), samples high-informativeness prompts by weighted sampling, and evolves them into new prompts that the solver trains on, in both offline and online settings.

Key results

• On Arena-Hard, DPO-eva raises gemma-2-9b-it win rate from 51.6% to 60.1%, and RLOO-eva from 52.6% to 62.4%, surpassing claude-3-opus and approaching gemini-1.5-pro.
• eva with offline SimPO reaches 60.7% Arena-Hard win rate from 52.3% after one iteration, outperforming training on new human prompts (54.6%).
• Replacing the reward-advantage informativeness with inverted advantage significantly hurts alignment gains, and uniform evolving gives 57.5% vs 60.1% for the default.

Measured effects reported in the paper

Surprising findings

• Inverting the advantage-based priority (favoring high-loss prompts) significantly hurts alignment gains, contrary to common hard-example prioritization intuitions.
• Training on evolved prompts only, without any new human prompts, matched or outperformed training with additional new UltraFeedback prompts.

Limitations

• Assumes a fixed oracle reward model; out-of-distribution prompts may need continual reward model training.
• Creator policy is not differentiable; evolving relies on an external prompt-rewriting method (Evol-Instruct style) rather than learned creation.
• Evaluation relies on Gemini-1.5 as generative scorer and benchmarks primarily on instruction following; reasoning not evaluated.

Experimental setup

Datasets: UltraFeedback

Benchmarks: Arena-Hard MT-Bench AlpacaEval 2.0

Metrics: win rate (%) MT-Bench rating length-controlled win rate

Baselines: RLOO OAIF DPO SPPO SimPO ORPO uniform evolving no-evolve active selection claude-3-opus gemini-1.5-pro

Builds on: DPO Evol-Instruct (WizardLM) prioritized experience replay RLOO minimax regret curriculum (PAIRED / PLR)

Closest papers by meaning

• Reward Is Enough: LLMs Are In-Context Reinforcement Learners (ICLR 2026 · similarity 0.90)
• LLM-as-a-Tutor: Policy-Aware Prompt Adaptation for Non-Verifiable RL (NeurIPS 2026 · similarity 0.89)
• LMRL Gym: Benchmarks for Multi-Turn Reinforcement Learning with Language Models (ICML 2025 · similarity 0.89)
• REvolve: Reward Evolution with Large Language Models using Human Feedback (ICLR 2025 · similarity 0.89)
• Behavior Injection: Preparing Language Models for Reinforcement Learning (NeurIPS 2025 · similarity 0.89)
• EvoLM: In Search of Lost Language Model Training Dynamics (NeurIPS 2025 · similarity 0.89)