Research map / Large language models / Alignment and preferences
Preference-grounded Token-level Guidance for Language Model Fine-tuning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's title only (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| token-level preference-grounded guidance instead of delayed RL feedback (RLPrompt) | prompt generation · SST-2 | accuracy: 90.5 -> 92.6 | Helped |
| token-level guidance instead of RLPrompt | prompt generation · AG News | accuracy: 79.7 -> 82.8 | Helped |
| MAX aggregation instead of summation (ablation) | prompt generation · AG News | accuracy: 82.8 -> 83.3 | Helped |
| MAX aggregation instead of summation (ablation) | prompt generation · Yelp P. | accuracy: 94.7 -> 94.8 | Helped |
| MAX aggregation instead of summation (ablation) | prompt generation · SST-2 | accuracy: 92.6 -> 91.2 | Hurt |
| reward-weighted MLE with learned guidance instead of supervised fine-tuning | T5-small · CNN/DM | ROUGE-L: 32.58 -> 38.17 | Helped |
| reward-weighted MLE with learned guidance instead of supervised fine-tuning | T5-small · XSum | ROUGE-1: 31.79 -> 33.62 | Helped |
| our guidance instead of PPO | T5-small · XSum | ROUGE-1: 23.18 -> 33.62 | Helped |
| average aggregation instead of summation (ablation) | summarization · CNN/DM | ROUGE-L: 37.93 -> 38.17 | Helped |
| MAX aggregation instead of average (ablation) | summarization · CNN/DM | ROUGE-L: 38.17 -> 37.26 | Hurt |
| soft minimum aggregation instead of average (ablation) | summarization · CNN/DM | ROUGE-1: 40.94 -> 40.78 | No clear effect |
| sequence-level preference guidance instead of token-level (ablation) | prompt and summarization tasks · SST-2, CNN/DM, XSum | accuracy / avg ROUGE | Hurt |
Closest papers by meaning
- Preference-Oriented Supervised Fine-Tuning: Favoring Target Model over Aligned Large Language Models (AAAI 2025 · similarity 0.88)
- Meta-Learning Preferences for Multilingual LLM Alignment (NeurIPS 2026 · similarity 0.88)
- Advantage-Guided Distillation for Preference Alignment in Small Language Models (ICLR 2025 · similarity 0.88)
- Spread Preference Annotation: Direct Preference Judgment for Efficient LLM Alignment (ICLR 2025 · similarity 0.88)
- Self-Supervised Alignment with Mutual Information: Learning to Follow Principles without Preference Labels (NeurIPS 2024 · similarity 0.88)
- Improved Representation Steering for Language Models (NeurIPS 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.