AI Paper Atlas

Research map / Large language models / Alignment and preferences

What's In My Human Feedback? Learning Interpretable Descriptions of Preference Data

Rajiv Movva, Smitha Milli, Sewon Min, Emma Pierson

ICLR 2026 · Large language models › Alignment and preferences · approach: Reward model design and preference models

Read the paperCheck your idea against 137,217 papers

AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Measured effects reported in the paper

What was changedWhereMeasuredEffect
restrict to 4 active SAE features per pair to predict preference instead of black-box reward modelWIMHF · 7 preference datasetsAUC: 0.672 vs 0.766 (oracle)Mixed
flip labels of top 1000 examples with the anti-refusal feature in ArenaLlama-3.2-3B reward model trained on Arena · RewardBench2 safetyaccuracy (%): 8.9 -> 46.2Improved
flip labels of anti-refusal examples in ArenaLlama-3.2-3B reward model trained on Arena · RewardBench2 non-safety propertiesaccuracyNo clear effect
personalize paragraphs-vs-lists feature coefficients per annotatorglobal preference model · Community Alignment held-out annotatorsheld-out AUC gain (%): up to +1.1% at k = 16Improved
actively sample examples with largest feature values instead of random samples for personalization (ablation)selective personalization · Community Alignment, low kheld-out AUCImproved
use SAE instead of top-variance embedding dimensions or PCA to build features (ablation)WIMHF · 7 datasetsmean fidelity: 0.20 (TopDims), 0.13 (PCA) -> 0.33 (SAE)Improved
increase SAE size M from 32 to 128 (ablation)WIMHF SAE · 7 datasetsfraction non-redundant high-fidelity features: 39.3% -> 29.2%Hurt
increase SAE size M beyond 32 (ablation)WIMHF SAE · 7 datasetsvalidation AUCNo clear effect
embed full prompt-response transcripts instead of responses only (ablation)WIMHF embeddings · preference predictionAUCNo clear effect
use Llama-3.1-8B reward model instead of Llama-3.2-3B (ablation)oracle reward model · Community Alignment held-out accuracyaccuracy (%): 68.6 -> 68.7No clear effect
use WIMHF instead of Inverse Constitutional AI to describe preferencespreference explanation · 5 datasets, top 10 featuressignificant features out of 50: 28 -> 43Improved
swap OpenAI embeddings for ModernBERT embeddings (ablation)WIMHF · HH-RLHFfeature similarity (qualitative)No clear effect

Closest papers by meaning

Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.

Open scime draft

Get the next scime Dispatch and a monthly field update

Free. We send a confirmation link first; every email has a one-click unsubscribe. Privacy