Research map / Large language models / Alignment and preferences
LLMdoctor: Token-Level Flow-Guided Preference Optimization for Efficient Test-Time Alignment of Large Language Models
Tiesunlong Shen, Rui Mao, Jin Wang, Heming Sun, Jian Zhang, Xuejie Zhang, Erik Cambria
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add subtrajectory balance loss (ablation) | LLMdoctor · HH-RLHF test | Win + 1/2 Tie (%) vs. DPO: 53.15 -> 61.00 | Improved |
| add subtrajectory balance loss (ablation) | LLMdoctor · HH-RLHF test | Diversity: 0.34 -> 0.47 | Improved |
| add value discrimination loss (ablation) | LLMdoctor · HH-RLHF test | Win + 1/2 Tie (%) vs. DPO: 58.23 -> 61.00 | Improved |
| add value discrimination loss (ablation) | LLMdoctor · HH-RLHF test | Diversity: 0.43 -> 0.47 | Improved |
| add reward sparsity threshold (ablation) | LLMdoctor · HH-RLHF test | Win + 1/2 Tie (%) vs. DPO: 56.58 -> 61.00 | Improved |
| add reward sparsity threshold (ablation) | LLMdoctor · HH-RLHF test | Diversity: 0.46 -> 0.47 | No clear effect |
| add flow-guided rewards (ablation) | LLMdoctor · HH-RLHF test | Win + 1/2 Tie (%) vs. DPO: 52.76 -> 61.00 | Improved |
| add flow-guided rewards (ablation) | LLMdoctor · HH-RLHF test | Diversity: 0.25 -> 0.47 | Improved |
| use LLMdoctor instead of GenARM as test-time guidance | LLaMA-7B-SFT · HH-RLHF test | Win + 1/2 Tie (%): 62.10 | Improved |
| use LLMdoctor test-time guidance instead of DPO training | LLaMA-7B-SFT · HH-RLHF test | Win + 1/2 Tie (%): 61.00 | Improved |
| use 7B doctor to guide 70B patient instead of DPO fine-tuning | Tulu2-SFT 70B · AlpacaEval 2 | LC win rate (%): 82.0 -> 82.5 | Improved |
| use ARGS instead of DPO | LLaMA-7B-SFT · HH-RLHF test | Win + 1/2 Tie (%): 26.24 | Hurt |
Closest papers by meaning
- Test-Time Preference Optimization: On-the-Fly Alignment via Iterative Textual Feedback (ICML 2025 · similarity 0.90)
- GenARM: Reward Guided Generation with Autoregressive Reward Model for Test-Time Alignment (ICLR 2025 · similarity 0.90)
- Reward-Augmented Data Enhances Direct Preference Alignment of LLMs (ICML 2025 · similarity 0.90)
- RMB: Comprehensively benchmarking reward models in LLM alignment (ICLR 2025 · similarity 0.89)
- Aligning to Thousands of Preferences via System Message Generalization (NeurIPS 2024 · similarity 0.89)
- TODO: Enhancing LLM Alignment with Ternary Preferences (ICLR 2025 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
