Research map / Large language models / AI and society
CollabLLM: From Passive Responders to Active Collaborators
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Trains LLMs to be proactive multiturn collaborators by optimizing Multiturn-aware Rewards estimated via forward simulation with an LLM user simulator, using SFT, DPO, and PPO on Llama-3.1-8B.
Problem
Standard LLM training (RLHF, next-turn rewards) optimizes single-turn responses, so models respond passively to ambiguous or open-ended requests and produce inefficient multiturn conversations.
Approach
Estimate each response's long-term value as the expected final reward (task success, token efficiency, LLM-judged interactivity) over simulated future conversations with an LLM user simulator (forward sampling with window w), then use these Multiturn-aware Rewards to build SFT/DPO data and to drive online RL, without a learned reward model.
Key results
- CollabLLM (Online DPO) improves task performance by 18.5% and interactivity by 46.3% on average over the best baselines, per LLM judges.
- In a user study with 201 MTurk participants, CollabLLM raises user satisfaction by 17.6% and reduces time spent by 10.4% versus Base.
- On MediumDocEdit-Chat, Online DPO raises ITR from 0.46 (Base) to 0.92 while BLEU goes from 32.2 to 36.8.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add proactive prompting with follow-up and clarification questions | Llama-3.1-8B-Instruct · MediumDocEdit-Chat | BLEU: 32.2% to 35.0% | Helped |
| Train with multiturn-aware rewards using Online DPO rather than Proactive Base | Llama-3.1-8B-Instruct · MediumDocEdit-Chat | BLEU: 35.0 -> 36.8 | Helped |
| Train with multiturn-aware rewards using Online DPO rather than Proactive Base | Llama-3.1-8B-Instruct · BigCodeBench-Chat | Pass Rate: 11.0 -> 13.0 | Helped |
| Train with multiturn-aware rewards using Online DPO rather than Proactive Base | Llama-3.1-8B-Instruct · MATH-Chat | Accuracy: 12.5 -> 16.5 | Helped |
| Use Online DPO rather than the base model | Llama-3.1-8B-Instruct · MediumDocEdit-Chat | ITR: 0.46 to 0.92 | Helped |
| Use Online DPO rather than the base model | Llama-3.1-8B-Instruct · MATH-Chat | Average token count (k): 1.03k | Helped |
| Use multiturn-aware rewards with forward sampling instead of immediate rewards (ablation) | CollabLLM · MediumDocEdit-Chat | BLEU | Helped |
| Increase forward sampling window size within the tested range (ablation) | CollabLLM · MediumDocEdit-Chat | BLEU | Helped |
| Use immediate helpfulness reward instead of multiturn-aware reward (ablation) | CollabLLM · MediumDocEdit-Chat | Average token count | Hurt |
| Use extrinsic-only reward instead of combined intrinsic and extrinsic multiturn-aware reward (ablation) | CollabLLM · MediumDocEdit-Chat | BLEU: 0.377 at w=1 | Helped |
| Use extrinsic-only reward instead of combined intrinsic and extrinsic multiturn-aware reward (ablation) | CollabLLM · MediumDocEdit-Chat | ITR | Hurt |
| Use intrinsic-only reward instead of combined intrinsic and extrinsic multiturn-aware reward (ablation) | CollabLLM · MediumDocEdit-Chat | ITR: 0.74 at w=1 | Helped |
Surprising findings
- Open-source models performed poorly as user simulators, often role-playing as assistants instead of users, so GPT-4o-mini was used.
- Proactive prompting alone gives only modest gains and can cause redundant clarification questions.
Limitations
- Relies on GPT-4o-mini as user simulator; open-source simulators performed poorly, and simulated users may not match real users.
- User study feedback notes CollabLLM can feel bland, lack up-to-date information, and needs effort to personalize output.
- Evaluation uses LLM judges for interactivity and accuracy, which may be biased.
Experimental setup
Datasets: MediumDocEdit-Chat BigCodeBench-Chat MATH-Chat BigCodeBench MATH Medium articles Abg-CoQA
Benchmarks: MediumDocEdit-Chat BigCodeBench-Chat MATH-Chat Abg-CoQA
Metrics: BLEU pass rate accuracy average token count interactivity (ITR) macro accuracy macro F1 user satisfaction rating time spent
Baselines: Llama-3.1-8B-Instruct Proactive Base SFT PPO Offline DPO GPT-4o ClarifyGPT STaR-GATE MTPO
Builds on: RLHF DPO PPO front-door adjustment
Closest papers by meaning
- ACC-Collab: An Actor-Critic Approach to Multi-Agent LLM Collaboration (ICLR 2025 · similarity 0.89)
- MINT: Evaluating LLMs in Multi-turn Interaction with Tools and Language Feedback (ICLR 2024 · similarity 0.88)
- ULTRAFEEDBACK: Boosting Language Models with Scaled AI Feedback (ICML 2024 · similarity 0.88)
- CollabBench: Benchmarking and Unleashing Collaborative Ability of LLMs with Diverse Players via Proactive Engagement (ICML 2026 · similarity 0.87)
- Collab: Controlled Decoding using Mixture of Agents for LLM Alignment (ICLR 2025 · similarity 0.87)
- Found in Conversation: LLMs Teach Themselves to Close the Multi-Turn Gap (NeurIPS 2026 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.