Research map / Large language models / Alignment and preferences
Principled Fine-tuning of LLMs from User-Edits: A Medley of Preference, Supervision, and Reward
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Theoretically analyzes fine-tuning LLMs from user-edit deployment logs via SFT, DPO, and cost-based RL, and proposes late ensembling of these policies with a UCB bandit at test time, evaluated on simulated email writing and summarization.
Problem
How to fine-tune LLM agents from naturally occurring user edits in deployment logs, which implicitly carry supervised, preference, and cost feedback, and adapt robustly when test-time users differ from training users.
Approach
Unifies three feedback types (SFT on edits, DPO on edit pairs, and cost-based RL with a pessimistic cost model) under a balance-equation user model, derives sample-complexity/suboptimality bounds showing trade-offs, and combines the resulting policies via early (loss-level) or late (bandit-level UCB) ensembling.
Key results
- LateEnsemble achieves the lowest worst-case max suboptimality (0.1586 on the main table) versus 0.1612 for EarlyEnsemble and 0.5772 for SFT, with Base at 0.7364.
- On summarization with a strong user, EarlyEnsemble reaches 0.2092 mean edit cost versus 0.5377 for SFT and 1.0790 for DPO, with Base at 0.9455.
- With a Llama-3.3-70B test-time user, LateEnsemble attains the best max suboptimality (0.0862), and the main trends carry over.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use early ensembling of SFT and DPO losses instead of either single-feedback method | Llama 3.1 8B Instruct · Summarization, strong and weak users | mean edit cost: 0.5377 ± 0.02 -> 0.2092 ± 0.09; 0.8267 ± 0.06 -> 0.3586 ± 0.01 | Helped |
| Use early ensembling of SFT and DPO losses rather than DPO | Llama 3.1 8B Instruct · Email writing, strong and weak users | mean edit cost: 0.3365 ± 0.00 -> 0.3438 ± 0.06; 0.3368 ± 0.01 -> 0.4864 ± 0.01 | Mixed |
| Use late ensembling with UCB instead of a single offline policy | Llama 3.1 8B Instruct · Max SubOpt across four settings | Max SubOpt: 0.8698 ± 0.11 -> 0.1586 ± 0.04 | Helped |
| Train with SFT rather than the base model | Llama 3.1 8B Instruct · Summarization and email writing, strong user | mean edit cost: 0.9455 ± 0.01 -> 0.5377 ± 0.02; 0.5108 ± 0.03 -> 0.4159 ± 0.05 | Helped |
| Train with DPO rather than SFT | Llama 3.1 8B Instruct · Summarization and email writing, weak user | mean edit cost: 0.9304 ± 0.19 -> 0.8267 ± 0.06; 0.4539 ± 0.03 -> 0.3368 ± 0.01 | Helped |
| Use early ensembling rather than either SFT or DPO | Llama 3.1 8B Instruct · Summarization, strong and weak users; email writing, strong user, transfer test | mean edit cost: 0.4890 ± 0.01 -> 0.1845 ± 0.02; 0.7698 ± 0.01 -> 0.2577 ± 0.02; 0.2955 ± 0.01 -> 0.2406 ± 0.03 | Helped |
Surprising findings
- DPO and EarlyEnsemble learned policies that tended to repeat text until max tokens, requiring a post-generation trimming step applied to all methods.
- EarlyEnsemble underperforms DPO on email writing, which the authors attribute to sensitivity to the lambda hyperparameter.
Limitations
- Experiments use simulated LLM users rather than real human users, and the paper defers human studies.
- Pessimistic cost-based RL is theoretically analyzed but not implemented empirically.
- Theory relies on balance-equation, realizability, and concentrability assumptions.
Experimental setup
Datasets: Gao et al. summarization and email writing article/preference data
Benchmarks: Gao et al. email writing Gao et al. summarization
Metrics: mean total edit distance per episode max suboptimality across settings
Baselines: Base SFT DPO EarlyEnsemble
Builds on: DPO Gao et al. user-edit simulation setup UCB
Closest papers by meaning
- Aligning LLMs by Predicting Preferences from User Writing Samples (ICML 2025 · similarity 0.89)
- PersonalLLM: Tailoring LLMs to Individual Preferences (ICLR 2025 · similarity 0.87)
- Unveiling the Secret Recipe: A Guide For Supervised Fine-Tuning Small LLMs (ICLR 2025 · similarity 0.87)
- Aligning Language Models with Demonstrated Feedback (ICLR 2025 · similarity 0.86)
- Accelerating Unbiased LLM Evaluation via Synthetic Feedback (ICML 2025 · similarity 0.86)
- Do LLMs Recognize Your Preferences? Evaluating Personalized Preference Following in LLMs (ICLR 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.