Research map / Large language models / Alignment and preferences
What's In My Human Feedback? Learning Interpretable Descriptions of Preference Data
Rajiv Movva, Smitha Milli, Sewon Min, Emma Pierson
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| restrict to 4 active SAE features per pair to predict preference instead of black-box reward model | WIMHF · 7 preference datasets | AUC: 0.672 vs 0.766 (oracle) | Mixed |
| flip labels of top 1000 examples with the anti-refusal feature in Arena | Llama-3.2-3B reward model trained on Arena · RewardBench2 safety | accuracy (%): 8.9 -> 46.2 | Improved |
| flip labels of anti-refusal examples in Arena | Llama-3.2-3B reward model trained on Arena · RewardBench2 non-safety properties | accuracy | No clear effect |
| personalize paragraphs-vs-lists feature coefficients per annotator | global preference model · Community Alignment held-out annotators | held-out AUC gain (%): up to +1.1% at k = 16 | Improved |
| actively sample examples with largest feature values instead of random samples for personalization (ablation) | selective personalization · Community Alignment, low k | held-out AUC | Improved |
| use SAE instead of top-variance embedding dimensions or PCA to build features (ablation) | WIMHF · 7 datasets | mean fidelity: 0.20 (TopDims), 0.13 (PCA) -> 0.33 (SAE) | Improved |
| increase SAE size M from 32 to 128 (ablation) | WIMHF SAE · 7 datasets | fraction non-redundant high-fidelity features: 39.3% -> 29.2% | Hurt |
| increase SAE size M beyond 32 (ablation) | WIMHF SAE · 7 datasets | validation AUC | No clear effect |
| embed full prompt-response transcripts instead of responses only (ablation) | WIMHF embeddings · preference prediction | AUC | No clear effect |
| use Llama-3.1-8B reward model instead of Llama-3.2-3B (ablation) | oracle reward model · Community Alignment held-out accuracy | accuracy (%): 68.6 -> 68.7 | No clear effect |
| use WIMHF instead of Inverse Constitutional AI to describe preferences | preference explanation · 5 datasets, top 10 features | significant features out of 50: 28 -> 43 | Improved |
| swap OpenAI embeddings for ModernBERT embeddings (ablation) | WIMHF · HH-RLHF | feature similarity (qualitative) | No clear effect |
Closest papers by meaning
- Improving Context-Aware Preference Modeling for Language Models (NeurIPS 2024 · similarity 0.91)
- Peering Through Preferences: Unraveling Feedback Acquisition for Aligning Large Language Models (ICLR 2024 · similarity 0.91)
- Eliciting Human Preferences with Language Models (ICLR 2025 · similarity 0.90)
- Maximizing Signal in Human-Model Preference Alignment (AAAI 2025 · similarity 0.90)
- Aligning Language Models Using Follow-up Likelihood as Reward Signal (AAAI 2025 · similarity 0.89)
- Limited Preference Data? Learning Better Reward Model with Latent Space Synthesis (NeurIPS 2025 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
