Research map / Large language models / Alignment and preferences
Interpreting Learned Feedback Patterns in Large Language Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Trains probes on sparse-autoencoder-condensed MLP activations of RLHF/DPO-fine-tuned small LLMs to estimate implicit feedback (VADER sentiment reward and DPO helpful/toxic preferences), and validates probe features against GPT-4 explanations.
Problem
It is unclear whether LLMs fine-tuned with RLHF accurately learn the preferences underlying human feedback data; the paper seeks to measure divergence between learned feedback patterns (LFPs) in activations and the true feedback.
Approach
Train linear/logistic probes on concatenated sparse autoencoder outputs from high-parameter-divergence MLP layers to predict the feedback signal implicit in activations over contrastive (positive/neutral/negative) inputs, then compare probe-identified features with GPT-4-labelled LFP-related features and ablate them.
Key results
- Probe on Pythia-160m reaches Kendall Tau 0.093 (p=0.014) with VADER rewards, while Pythia-70m (p=0.26) and GPT-Neo-125m (p=0.48) are not significant.
- Logistic regression probes classify implicit DPO feedback in activations with 99.80%-100.00% accuracy across the four models and two DPO tasks.
- GPT-4-labelled LFP-related features are active for 13.9%-19.7% of strong-positive-feedback inputs versus 4.1%-9.1% for the average feature.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use logistic regression on condensed activations to predict binary feedback | DPO-fine-tuned probes · Anthropic HH-RLHF and toxic-dpo test sets | probe accuracy: ≥99.80% | Helped |
| Use condensed activations to predict token-level VADER reward | Pythia-160m probe · VADER | Kendall Tau correlation: 0.093 | Helped |
| Use condensed activations to predict token-level VADER reward | Pythia-70m and GPT-Neo-125m probes · VADER | Kendall Tau correlation: 0.042; 0.023 | No clear effect |
| Zero-ablate GPT-4-identified LFP-related features (ablation) | Fine-tuned model · VADER, Pythia-70m | average reward: 2.07 -> 1.96 | Hurt |
| Zero-ablate GPT-4-identified LFP-related features (ablation) | Fine-tuned model · VADER, Pythia-160m | average reward: 1.95 -> 1.69 | Hurt |
| Zero-ablate GPT-4-identified LFP-related features (ablation) | Fine-tuned model · VADER, GPT-Neo-125m | average reward: 1.43 -> 1.43 | No clear effect |
| Tie encoder and decoder weights rather than leave them untied (ablation) | Sparse autoencoder · Pythia-160m, 1 training epoch | reconstruction loss: 0.059 -> 0.053 | Helped |
| Tie encoder and decoder weights rather than leave them untied (ablation) | Sparse autoencoder · Pythia-70m, 1 training epoch | reconstruction loss: 0.036 -> 0.030 | Helped |
| Select lowest layers instead of highest-divergence layers (ablation) | Sparse autoencoder · Pythia-160m, 1 training epoch | reconstruction loss: 0.053 -> 0.023 | Helped |
| Select lowest layers instead of highest-divergence layers (ablation) | Sparse autoencoder · Pythia-70m, 1 training epoch | reconstruction loss: 0.036 -> 0.021 | Helped |
Surprising findings
- Probe accuracy on the DPO classification tasks is near-perfect, yet the fine-grained VADER correlation is weak for two of three models.
- Ablating GPT-4-identified LFP features reduced performance for Pythia-160m but left GPT-Neo-125m unchanged.
Limitations
- Sparse autoencoders may learn feature compositions not present in the raw activations, so probe features may not be faithful.
- Method does not provide a mechanistic explanation of how LFP features relate or affect expected feedback.
- Only small models (70M-2B parameters) and simple tasks were studied; VADER correlations are weak for most models.
Experimental setup
Datasets: IMDb VADER lexicon Anthropic HH-RLHF toxic-dpo
Benchmarks: VADER lexicon token rewards Anthropic HH-RLHF preference classification toxic-dpo preference classification
Metrics: Kendall Tau correlation classification accuracy p-value average reward feature activation frequency sign accuracy
Baselines: untrained linear regression baseline
Builds on: Proximal Policy Optimization Direct Preference Optimization sparse autoencoders VADER
Closest papers by meaning
- A Critical Evaluation of AI Feedback for Aligning Large Language Models (NeurIPS 2024 · similarity 0.91)
- Sequence to Sequence Reward Modeling: Improving RLHF by Language Feedback (AAAI 2025 · similarity 0.90)
- RLVF: Learning from Verbal Feedback without Overgeneralization (ICML 2024 · similarity 0.89)
- Peering Through Preferences: Unraveling Feedback Acquisition for Aligning Large Language Models (ICLR 2024 · similarity 0.89)
- Reshaping Reasoning in LLMs: A Theoretical Analysis of RL Training Dynamics through Pattern Selection (ICLR 2026 · similarity 0.89)
- Policy Improvement using Language Feedback Models (NeurIPS 2024 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.