Research map / Large language models / Alignment and preferences
TIS-DPO: Token-level Importance Sampling for Direct Preference Optimization With Estimated Weights
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes TIS-DPO, a token-level importance-sampling extension of DPO that weights each token by an estimated reward from contrastive LLMs (built via prompts, SFT, or DPO), improving harmlessness, helpfulness, and summarization alignment.
Problem
DPO treats the whole response as a single bandit arm and ignores differences in token importance, which can reduce optimization efficiency and final alignment quality.
Approach
Derive a token-level PPO objective with importance sampling from an ideal dataset where each token has equal expected reward, yielding a DPO variant with per-token weights; estimate weights from log-probability differences between a positive and negative contrastive LLM, built by contrastive prompting, separate SFT on winning/losing responses, or forward/reverse DPO.
Key results
- On PKU-SafeRLHF with LLaMA2-7B, TIS-DPO(D) reaches 96.7% Llama-Guard safety and 79.3% GPT-4 win rate versus DPO, versus 74.4% safety for DPO.
- On Anthropic-HH with Mistral-7B, TIS-DPO(D) achieves 92.6% Llama-Guard safety and 85.4% win rate, compared with 63.3% safety for DPO.
- TIS-DPO(D) and TIS-DPO(S) improve Llama-Guard safety by 20.0 and 26.1 points over the previous best baseline, respectively.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use DPO-based contrastive models to estimate token weights instead of standard DPO | LLaMA2-7B · PKU-SafeRLHF | Llama-Guard safety: 74.4% -> 96.7% | Helped |
| Use DPO-based contrastive models to estimate token weights instead of standard DPO | LLaMA2-7B · Anthropic-HH | Beaver-Reward Help.: 8.4 -> 9.2 | Helped |
| Use SFT-based contrastive models to estimate token weights instead of standard DPO | LLaMA2-7B · PKU-SafeRLHF | Llama-Guard safety: 74.4% -> 89.6% | Helped |
| Use prompt-based contrastive weight estimation instead of standard DPO | Mistral-7B · PKU-SafeRLHF | Llama-Guard safety: 81.2% -> 80.1% | Hurt |
| Replace estimated token weights with random weights (ablation) | TIS-DPO(D) · PKU-SafeRLHF | Llama-Guard safety: 96.7% -> 21% | Hurt |
| Replace estimated token weights with random weights (ablation) | TIS-DPO(D) · PKU-SafeRLHF | Beaver-Cost Harm: 0.1 -> 9.2 | Hurt |
| Replace estimated token weights with equal weights (ablation) | TIS-DPO(D) · PKU-SafeRLHF | Llama-Guard safety: 96.7% -> 74.9% | Hurt |
| Replace estimated token weights with equal weights (ablation) | TIS-DPO(D) · PKU-SafeRLHF | Beaver-Cost Harm: 0.1 -> 5.8 | Hurt |
| Remove the eta term (ablation) | TIS-DPO(D) · PKU-SafeRLHF | Llama-Guard safety: 96.7% -> 95.3% | Hurt |
| Remove the eta term (ablation) | TIS-DPO(D) · PKU-SafeRLHF | MT-bench: 4.3 -> 4.3 | No clear effect |
| Use contrastive prompts with LLM-generated data instead of the original data distribution | TIS-DPO(P) · PKU-SafeRLHF | Llama-Guard safety: 49.8% -> 68.3% | Helped |
| Use DPO-based contrastive models rather than prompt-based contrastive models | TIS-DPO · PKU-SafeRLHF with LLM-generated data | Llama-Guard safety: 68.3% -> 81.3% | Helped |
Surprising findings
- Prompt-based contrastive weight estimation (TIS-DPO(P)) gives limited gains on original datasets and sometimes underperforms plain DPO, attributed to mismatch between dataset and LLM output distribution.
- Random token weights performed far worse than DPO-level baselines (21% safety on PKU-SafeRLHF ablation), indicating weight quality is critical.
Limitations
- Prompt-based weight estimation depends on distribution match between dataset and model outputs.
- Weight estimates are noisy, as acknowledged by the authors' visualizations.
- Evaluation relies largely on GPT-4 win rates and automatic safety judges rather than human evaluation.
Experimental setup
Datasets: PKU-SafeRLHF Anthropic-HH TL;DR
Benchmarks: PKU-SafeRLHF Anthropic-HH AdvBench JailbreakBench Alpaca MT-bench TL;DR
Metrics: Llama-Guard safety percentage Beaver-Cost harmfulness score Beaver-Reward helpfulness score MT-bench score GPT-4 win rate
Baselines: DPO PPO IPO KTO TDPO RLCD
Builds on: DPO TDPO RLCD Regret Preference Model
Closest papers by meaning
- Less is More: Improving LLM Alignment via Preference Data Selection (NeurIPS 2025 · similarity 0.92)
- Cal-DPO: Calibrated Direct Preference Optimization for Language Model Alignment (NeurIPS 2024 · similarity 0.92)
- What Matters in Data for DPO? (NeurIPS 2025 · similarity 0.91)
- Token-level Direct Preference Optimization (ICML 2024 · similarity 0.91)
- Noise Contrastive Alignment of Language Models with Explicit Rewards (NeurIPS 2024 · similarity 0.91)
- The Crucial Role of Samplers in Online Direct Preference Optimization (ICLR 2025 · similarity 0.91)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.