Research map / Large language models / Alignment and preferences
Would I Lie To You? Inference Time Alignment of Language Models using Direct Preference Heads
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Direct Preference Heads (DPH), an auxiliary reward head trained jointly with cDPO on a 551M LM so that rewards rerank sampled candidates at inference without changing the LM head's output distribution.
Problem
RLHF and DPO alignment can degrade reasoning ability and induce hallucination in small language models, since they directly alter the output token distribution.
Approach
Attach a learnable reward head to the final-token hidden state of the LM and train it with separable or contrastive DPH losses (theoretically linked to cDPO) jointly with cDPO on the LM logits, so that alignment rewards are used only to select candidates at inference time while the generation distribution is largely preserved.
Key results
- Ours_DPH reaches 60.21 average on the GPT4All suite versus 53.04 for Ours_SFT, 57.05 for Ours_DPO, and 52.99 for TinyLlama (1.1B).
- Ours_DPH reaches 62.5 weighted average on RACE, about 3.5% above GPT-1 (59.0) and above LLaMA 7B (51.0).
- On GLUE (w/o WNLI), Ours_DPH scores 75.0 vs 72.7 for Ours_DPO and 68.2 for Ours_SFT (w/ WNLI).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| select candidates using DPH rewards rather than log probabilities | 551M LM · GLUE w/o WNLI | score: 77.0 -> 78.6 | Helped |
| select candidates using DPH rewards rather than log probabilities | 551M LM · GPT4All | average accuracy: 57.05 -> 60.21 | Helped |
| select candidates using DPH rewards rather than log probabilities | 551M LM · RACE test | weighted average accuracy: 61.6 -> 62.5 | Helped |
| use SwiGLU FFN pooling instead of identity pooling with Separable DPH (ablation) | DPH · RACE validation | score: 56.54 -> 57.60 | Helped |
| use SwiGLU FFN pooling instead of identity pooling with Separable DPH (ablation) | DPH · GPT4All validation | score: 56.86 -> 57.14 | No clear effect |
| use Contrastive instead of Separable DPH with SwiGLU pooling (ablation) | DPH · GPT4All validation | score: 57.14 -> 58.28 | Helped |
| use Contrastive instead of Separable DPH with SwiGLU pooling (ablation) | DPH · GLUE validation | score: 75.19 -> 74.04 | Hurt |
| use Contrastive instead of Separable DPH with SwiGLU pooling (ablation) | DPH · RACE validation | score: 57.60 -> 55.95 | Hurt |
| train a GLUE-specific head rather than use the baseline head (ablation) | DPH checkpoint · GLUE validation | score: 76.12 -> 76.36 | Helped |
| train a GPT4All-specific head rather than use the baseline head (ablation) | DPH checkpoint · GPT4All validation | score: 60.19 -> 60.29 | No clear effect |
| apply DPH selection rather than log-probability selection (ablation) | Qwen1.5-0.5B · GLUE validation | score: 41.94 -> 45.69 | Helped |
| apply DPH selection rather than log-probability selection (ablation) | Qwen1.5-0.5B · RACE validation | score: 51.38 -> 41.21 | Hurt |
Surprising findings
- Qwen1.5 chat variants degrade on log-probability GLUE/RACE scores relative to base models at 0.5B, supporting the claim that alignment can hurt small-model reasoning.
- Contrastive DPH outperforms separable DPH on sentence-completion tasks such as HellaSwag, WinoGrande and PIQA, despite separable being best on GLUE and RACE.
Limitations
- DPH can only detect harmful or incorrect outputs at inference time and does not prevent generation of them.
- Results rely on a small in-house 551M model and a few Qwen1.5 variants, limiting generality to larger models.
- Preference pairs for GLUE, RACE, GPT4All, MMLU and SQuAD were synthesized from labeled answers rather than human preference data.
Experimental setup
Datasets: GLUE GPT4All RACE Alpaca OpenOrca UltraFeedback MMLU SQuAD V2 TinyStories CNN-Dailymail CoQA The Pile
Benchmarks: GLUE GPT4All RACE HellaSwag OpenBookQA WinoGrande ARC BoolQ PIQA
Metrics: accuracy macro-average score Matthews/F1/Pearson-Spearman correlation
Baselines: SFT DPO TinyLlama Pythia-1.0B Pythia-1.4B GPT-1 BERT-Base BERT-Large LLaMA 7B LLaMA 13B Qwen1.5-0.5B-Chat Qwen1.5-1.8B-Chat
Builds on: Direct Preference Optimization Conservative DPO RLHF Prior Regularization Transformer-XL
Closest papers by meaning
- Online Preference Alignment for Language Models via Count-based Exploration (ICLR 2025 · similarity 0.91)
- Towards Efficient Exact Optimization of Language Model Alignment (ICML 2024 · similarity 0.91)
- Active Preference Learning for Large Language Models (ICML 2024 · similarity 0.91)
- Preference Ranking Optimization for Human Alignment (AAAI 2024 · similarity 0.90)
- Earlier Tokens Contribute More: Learning Direct Preference Optimization From Temporal Decay Perspective (ICLR 2025 · similarity 0.90)
- Doubly Robust Alignment for Large Language Models (NeurIPS 2025 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.