Research map / Large language models / Alignment and preferences

RLVF: Learning from Verbal Feedback without Overgeneralization

ICML 2024 · Large language models › Alignment and preferences · approach: LLM-as-judge and feedback data · 1 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes C3PO, which turns a single piece of verbal feedback into synthetic in-scope preference pairs plus out-of-scope and near-scope regularization data, so an LLM adopts feedback where relevant without overgeneralizing.

Problem

Incorporating high-level verbal feedback into LLMs (e.g., "Don't use emojis when drafting emails to my boss") tends to overgeneralize: the feedback also changes behavior on prompts where it should not apply, while prompting with feedback is costly and unreliable at scale.

Approach

Use a strong LLM to generate categories and in-scope, near-scope, and out-of-scope prompts from a feedback sentence, create two-policy preference pairs (original vs. feedback-revised responses), and train with DPO on in-scope pairs plus SFT regularization on out-of-scope and near-scope prompts.

Key results

• C3PO achieves S_overall of 0.7065 versus 0.587 for In-Context and 0.4045 for DPO, with S_in 0.563 and S_out 0.150.
• C3PO reduces out-of-scope behavior change to 0.150 from 0.503 for In-Context, in line with the abstract's claim of a 30% reduction in overgeneralization.
• Merging LoRA parameters from two separately trained feedbacks shows virtually no degradation in in-scope adherence on average across 10 pairs.

Measured effects reported in the paper

Surprising findings

• Chain-of-thought reasoning about feedback applicability (In-Context + CoT) substantially lowers in-scope adherence (0.402 vs 0.677) without improving overall score.
• Full knowledge distillation on out-of-scope prompts, instead of maximum likelihood on baseline samples, substantially impairs in-scope adherence.

Limitations

• Relies on GPT-4 for category and prompt generation and on the base model's ability to correctly interpret feedback.
• Heuristic scoring covers only 14 human-written feedbacks; the rest depend on GPT-4 judgments.
• Stronger distillation-based constraints hurt in-scope adherence, and the best constraint form remains open.

Experimental setup

Datasets: Open Instruction Generalist (OIG) GPT-4-generated synthetic prompts and feedback

Benchmarks: custom feedback dataset of 100 pieces of feedback with held-out in-scope, out-of-scope, and near-scope test prompts

Metrics: in-scope feedback adherence S_in out-of-scope behavior change S_out overall score S_overall

Baselines: In-Context In-Context + CoT SCD SCD + Negatives DPO

Builds on: direct preference optimization supervised context distillation constitutional AI context distillation

Closest papers by meaning

• Interpreting Learned Feedback Patterns in Large Language Models (NeurIPS 2024 · similarity 0.89)
• MA-RLHF: Reinforcement Learning from Human Feedback with Macro Actions (ICLR 2025 · similarity 0.89)
• Peering Through Preferences: Unraveling Feedback Acquisition for Aligning Large Language Models (ICLR 2024 · similarity 0.89)
• SELF-EVOLVED REWARD LEARNING FOR LLMS (ICLR 2025 · similarity 0.89)
• Explicit Preference Optimization: No Need for an Implicit Reward Model (ICML 2025 · similarity 0.89)
• RLAIF vs. RLHF: Scaling Reinforcement Learning from Human Feedback with AI Feedback (ICML 2024 · similarity 0.89)