Research map / Large language models / Alignment and preferences

Aligning Language Models Using Follow-up Likelihood as Reward Signal

AAAI 2025 · Large language models › Alignment and preferences · approach: Active and token-level preference learning · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes Follow-up Likelihood as Reward (FLR), which scores responses by the LM's likelihood of curated positive vs. negative user follow-up utterances, and uses it to auto-annotate preference pairs for DPO/KTO alignment, with optional fine-tuning of the scorer on natural-language feedback.

Problem

Reward modeling and preference annotation for LLM helpfulness typically require costly human or GPT-4 annotations or large trained reward models; the paper seeks an annotation-free signal of response quality.

Approach

Use the conditional log-likelihood of manually curated positive and negative user follow-up utterances (decomposed into understanding, engagingness, instruction-following) given a prompt-response pair as a reward, mine preference pairs from the policy's own online samples, and fine-tune the scorer LM on rewritten natural-language feedback to sharpen its follow-up likelihoods.

Key results

• FLR-FT (Llama-3-8B-Instruct) reaches 71.02% average pairwise accuracy and 0.577 average Pearson on reward benchmarks, ranking third on pairwise and first on rating-based benchmarks among compared reward models.
• Llama-3-8B-Instruct + FLR DPO improves AlpacaEval 2 length-controlled win rate from 28.72 to 33.17 without any external annotations; FLR-FT DPO reaches 34.22.
• ArmoRM (trained on about 1M annotations) remains best on Alpaca-Eval (41.40 LC) and Arena-Hard (28.4), with the best FLR variant at 35.18 and 24.8 respectively.

Measured effects reported in the paper

Surprising findings

• Individual follow-ups can outperform the full ensemble; the negative understanding follow-up "What are you trying to say?" alone reaches 66.6% average pairwise accuracy in the reported individual-utterance analysis.
• FLR gains are model-dependent: Qwen2-7B-Instruct + FLR DPO shows little or no AlpacaEval gain (28.78 vs 29.74 base), while FLR-FT improves it.

Limitations

• Follow-up utterances are manually curated (30 per category sets of 10 positive/10 negative each across three categories), and automatic search for better follow-ups is left to future work.
• Performance gap remains against large-scale annotated reward models such as ArmoRM on alignment benchmarks.
• Effectiveness is sensitive to the underlying LM and training method, so gains are not uniform across models.

Experimental setup

Datasets: UltraFeedback Nectar HH-RLHF BeaverTails SHP Alpaca-Farm MT-Bench RewardBench Preference Bench Feedback Bench FLASK HelpSteer

Benchmarks: HH-RLHF BeaverTails SHP Alpaca-Farm MT-Bench Pairwise RewardBench Chat Easy RewardBench Chat Hard Preference Bench Feedback Bench FLASK MT-Bench Rating HelpSteer

Metrics: accuracy Pearson correlation Spearman correlation Kendall tau length-controlled win rate win rate

Baselines: Prometheus-7B GPT-3.5-Turbo Starling-RM-7B-alpha Oasst-Pythia-6.9B PairRM ArmoRM-Llama3-8B-v0.1 Direct log-probability scoring (Tulu-2-dpo-7b, Qwen2-7B-Instruct, Zephyr-7b-alph KTO

Builds on: DPO KTO LLM-as-a-Judge self-rewarding language models LLaMA-Factory

Closest papers by meaning

• Transforming and Combining Rewards for Aligning Large Language Models (ICML 2024 · similarity 0.91)
• Data with High and Consistent Preference Difference Are Better for Reward Model (AAAI 2025 · similarity 0.91)
• Sequence to Sequence Reward Modeling: Improving RLHF by Language Feedback (AAAI 2025 · similarity 0.90)
• Reward-Augmented Data Enhances Direct Preference Alignment of LLMs (ICML 2025 · similarity 0.90)
• Sequential Preference Optimization: Multi-Dimensional Preference Alignment with Implicit Reward Modeling (AAAI 2025 · similarity 0.90)
• Improving Context-Aware Preference Modeling for Language Models (NeurIPS 2024 · similarity 0.90)