Research map / Large language models / Alignment and preferences

SEAL: Systematic Error Analysis for Value ALignment

AAAI 2025 · Large language models › Alignment and preferences · approach: Reward modeling for diverse preferences · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes SEAL, a set of metrics (feature imprint, alignment resistance, alignment robustness) to audit how an RLHF reward model trained on Anthropic/hh-rlhf learns value and spoiler features, using GPT-4-turbo labels and OpenAssistant RMs.

Problem

Internal mechanisms of RLHF are poorly understood; practitioners lack systematic ways to tell whether reward models learn intended values, resist human preferences, or are sensitive to confounding stylistic features in alignment datasets.

Approach

Featurize each dialogue of the alignment dataset with an LM labeler into target (helpfulness, harmlessness) and spoiler features, then regress pre- and post-alignment RM rewards and reward shifts on these features to quantify imprints; define alignment resistance as the fraction of pairs where the RM disagrees with human labels; and measure robustness by LM-rewriting entries and regressing misalig

Key results

• Post-D RM alignment score is 0.74 vs 0.57 for pre-D RM; 26% alignment resistance on Anthropic/hh-rlhf.
• 8% of pairs aligned pre-D become misaligned post-D (reversal regime).
• Harmlessness reward rises from -0.85 (pre-D) to 2.09 (post-D); rewriting 1% of the data lowers alignment to 0.71.

Measured effects reported in the paper

Surprising findings

• Alignment resistance concentrates on pairs where the LM-labeler also disagrees with human labels (86% agreement on stayed-aligned vs 34% on stayed-misaligned).
• Positive-sentiment rewrites increase misalignment odds, showing sensitivity to spoiler features.

Limitations

• Feature labels from a single LM-labeler are unstable across models; sentiment and eloquence labels largely uncorrelated across labelers.
• Spoiler features are not systematically identified or defined.
• Helpfulness/harmlessness labels are treated as an approximation, not ground truth.

Experimental setup

Datasets: Anthropic/hh-rlhf web-gpt summarize-from-feedback synthetic-instruct-gptj-pairwise

Benchmarks: Anthropic/hh-rlhf

Metrics: feature imprint (regression coefficients) alignment score / alignment resistance reward shift angle robustness score (logistic regression coefficients) LM-labeler agreement rate

Baselines: pre-D reward model (OpenAssistant RM trained on semantic datasets)

Builds on: RLHF Anthropic/hh-rlhf BGE-M3

Closest papers by meaning

• Data with High and Consistent Preference Difference Are Better for Reward Model (AAAI 2025 · similarity 0.93)
• Doubly Robust Alignment for Large Language Models (NeurIPS 2025 · similarity 0.92)
• Linear Alignment: A Closed-form Solution for Aligning Human Preferences without Tuning and Feedback (ICML 2024 · similarity 0.92)
• RLTHF: Targeted Human Feedback for LLM Alignment (ICML 2025 · similarity 0.92)
• Reward-Augmented Data Enhances Direct Preference Alignment of LLMs (ICML 2025 · similarity 0.92)
• RMB: Comprehensively benchmarking reward models in LLM alignment (ICLR 2025 · similarity 0.91)