Research map / Large language models / Alignment and preferences
Coactive Learning for Large Language Models using Implicit User Feedback
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes coactive learning with CoRLL, which trains LLMs by DPO on implicit user edit feedback (edited output preferred over the model's output), and evaluates it against dueling RLHF on TL;DR, HH, and IMDB.
Problem
Training LLMs from weak implicit feedback (user edits of model outputs) that is not gold-standard, rather than explicitly labeled pairwise preferences as in RLHF.
Approach
Treats user edits ȳ of an LLM output y as a preference ȳ ≻ y in a coactive learning framework, proves a regret bound reducing to a pairwise classification learner, and instantiates it as CoRLL, which updates the policy with DPO using sampled argmax approximations.
Key results
- On the summarization task, CoRLL with minimally informative coactive feedback reaches a GPT-4 winrate of 0.6735 vs. the base model and similarity 0.8530, with noisy feedback giving comparable 0.6730 winrate.
- CoRLL with minimally informative feedback achieves a better reward-vs-KL tradeoff than dueling-feedback DPO on both summarization and helpful assistant tasks.
- On IMDB, CoRLL remains effective even at feedback quality alpha=0.1 and with ~40% mislabeled preferences, and increasing the argmax approximation k beyond 3 gives little extra gain.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use minimally informative coactive feedback instead of dueling feedback | CoRLL · Reddit TL;DR | winrate: 0.5595 -> 0.6735 | Helped |
| use minimally informative coactive feedback instead of dueling feedback | CoRLL · Reddit TL;DR | similarity: 0.7505 -> 0.8530 | Helped |
| use minimally informative coactive feedback instead of dueling feedback | CoRLL · Helpful Assistant | winrate: 0.5440 -> 0.5875 | Helped |
| use minimally informative coactive feedback instead of dueling feedback | CoRLL · Helpful Assistant | similarity: 0.8280 -> 0.9240 | Helped |
| use noisy rather than noise-free minimally informative coactive feedback (ablation) | CoRLL · Reddit TL;DR | winrate: 0.6735 -> 0.6730 | No clear effect |
| use noisy rather than noise-free minimally informative coactive feedback (ablation) | CoRLL · Helpful Assistant | winrate: 0.5875 -> 0.5894 | No clear effect |
| use IPO instead of DPO for policy updates (ablation) | CoRLL · Reddit TL;DR | reward | Hurt |
| increase the number k of policy samples used to approximate the argmax (ablation) | CoRLL · IMDB | reward | Helped |
| increase coactive feedback quality (ablation) | CoRLL · IMDB | reward | Helped |
| inject mislabeled preference feedback (ablation) | CoRLL · IMDB | reward | No clear effect |
| edit only the last 5 tokens to produce coactive feedback (ablation) | CoRLL · Reddit TL;DR | reward | Helped |
Surprising findings
- Coactive feedback learns faster and achieves a better reward/KL tradeoff than conventional dueling-feedback DPO despite being weaker and noisier.
- CoRLL remains effective even when the mislabeling rate reaches roughly 35-45%.
Limitations
- Feedback is simulated from an expert policy rather than collected from real human editors.
- Argmax approximated by sampling k responses, with k=1 used in large-scale experiments for tractability.
- Coactive and dueling feedback are not directly comparable in data collection cost.
Experimental setup
Datasets: Reddit TL;DR Anthropic Helpful and Harmless IMDB
Benchmarks: Reddit TL;DR Anthropic Helpful and Harmless IMDB
Metrics: implicit expert reward GPT-4 winrate GPT-4 similarity to expert KL divergence error rate of noisy preferences
Baselines: Dueling DPO IPO RLHF (dueling feedback)
Builds on: Direct Preference Optimization coactive learning (Shivaswamy & Joachims) RLHF Bradley-Terry model
Closest papers by meaning
- Enabling Lanuguage Models to Implicitly Learn Self-Improvement (ICLR 2024 · similarity 0.85)
- Teaching Models to Improve on Tape (AAAI 2025 · similarity 0.85)
- Cross-model Control: Improving Multiple Large Language Models in One-time Training (NeurIPS 2024 · similarity 0.85)
- Co-rewarding: Stable Self-supervised RL for Eliciting Reasoning in Large Language Models (ICLR 2026 · similarity 0.85)
- ULTRAFEEDBACK: Boosting Language Models with Scaled AI Feedback (ICML 2024 · similarity 0.85)
- Principled Fine-tuning of LLMs from User-Edits: A Medley of Preference, Supervision, and Reward (NeurIPS 2025 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.