Research map / Large language models / Alignment and preferences
TokenRatio: Principled Token-Level Preference Optimization via Ratio Matching
Truong Nguyen, Tien-Phat Nguyen, Linh Ngo Van, Duy Minh Ho Nguyen, Khoa D Doan, Trung Le
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| token-level Bregman ratio matching (TBPO-Q) instead of sequence-level DPO | DPO, Mistral 7B v0.1 · Open LLM Leaderboard average | average score: 59.30 -> 60.95 | Improved |
| token-level Bregman ratio matching (TBPO-A) instead of sequence-level DPO | DPO, Llama 3 8B · Open LLM Leaderboard average | average score: 66.24 -> 70.50 | Improved |
| TBPO-Q instead of DPO | DPO, Mistral 7B v0.1 · GSM8K | accuracy: 34.87 -> 39.34 | Improved |
| TBPO-Q instead of DPO | DPO, Mistral 7B v0.1 · TruthfulQA | accuracy: 42.30 -> 44.47 | Improved |
| TBPO-A instead of TDPO | TDPO, Llama 3 8B · ARC | accuracy: 59.04 -> 64.08 | Improved |
| learned baseline head (Q) versus KL-based baseline (A) for the state weight | TBPO · Open LLM Leaderboard average | average score: 60.95 vs 60.73 (Mistral); 70.38 vs 70.50 (Llama 3) | No clear effect |
| TBPO instead of baselines for pairwise judging | TDPO, DPO, TIS-DPO, BPO · MT-Bench | pairwise win rate | Improved |
| TBPO instead of baselines | DPO and token-level baselines, Llama 3 8B · Anthropic HH-RLHF | LC win rate | Improved |
| TBPO instead of DPO-style methods | DPO, TDPO, BPO · UltraFeedback generation diversity | predictive entropy, Distinct-1, Self-BLEU | Improved |
| TIS-DPO token-level importance sampling | DPO · UltraFeedback generation diversity | predictive entropy, Self-BLEU | Hurt |
| TDPO token-wise decomposition instead of DPO | DPO, Mistral 7B v0.1 · GSM8K | accuracy: 34.87 -> 28.65 | Hurt |
Closest papers by meaning
- ConfPO: Exploiting Policy Model Confidence for Critical Token Selection in Preference Optimization (ICML 2025 · similarity 0.91)
- Earlier Tokens Contribute More: Learning Direct Preference Optimization From Temporal Decay Perspective (ICLR 2025 · similarity 0.91)
- TIS-DPO: Token-level Importance Sampling for Direct Preference Optimization With Estimated Weights (ICLR 2025 · similarity 0.91)
- Cal-DPO: Calibrated Direct Preference Optimization for Language Model Alignment (NeurIPS 2024 · similarity 0.91)
- Test-Time Preference Optimization: On-the-Fly Alignment via Iterative Textual Feedback (ICML 2025 · similarity 0.90)
- Token-level Direct Preference Optimization (ICML 2024 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
