Research map / Large language models / Retrieval and knowledge
Modeling Future Conversation Turns to Teach LLMs to Ask Clarifying Questions
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes double-turn preference labeling that scores clarifying questions by the answers an LLM gives after simulated user clarifications, training LLMs to ask clarifying questions for open-domain QA.
Problem
LLMs often presuppose one interpretation of ambiguous requests instead of asking clarifying questions, partly because single-turn preference labeling cannot assess the utility of a clarifying question.
Approach
Assign preference labels to clarifying questions and direct answers by simulating the user's clarifying answer and the model's subsequent answer, aggregating correctness across multiple annotated answers (one simulated user per gold answer), and training with DPO; the system also learns when to answer directly.
Key results
- Clarify DPO w/ Match achieves 32.8 / 36.5 / 33.9 All-set Answer F1 on Llama2 / Llama3 / Gemma versus 25.9 / 31.5 / 28.8 for Clarify SFT (Llama2 and Gemma numbers as reported in Table 1).
- Clarify-or-Direct-Ans DPO on Llama2 reaches 24.3 Answer F1 with 1.56 turns, compared with 21.3 for its SFT version.
- Joint single-model merge loses performance (Llama2 All F1 28.7 vs 32.8 for separate models).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use double-turn answer matching instead of single-turn reward-model ranking | Clarify DPO · AmbigQA Full | Answer F1: 28.3 -> 32.8 | Helped |
| Use double-turn answer matching instead of single-turn reward-model ranking | Clarify DPO · AmbigQA Full | Answer F1: 30.9 -> 36.5 | Helped |
| Use double-turn answer matching instead of single-turn reward-model ranking | Clarify DPO · AmbigQA Full | Answer F1: 29.5 -> 33.9 | Helped |
| Add clarifying-question response capability rather than directly answering only | Llama2-7B Direct-Ans SFT · AmbigQA Full | Answer F1: 21.1 -> 32.8 | Helped |
| Add clarifying-question response capability rather than directly answering only | Llama3-8B Direct-Ans SFT · AmbigQA Full | Answer F1: 24.8 -> 36.5 | Helped |
| Add clarifying-question response capability rather than directly answering only | Gemma-7B Direct-Ans SFT · AmbigQA Full | Answer F1: 21.1 -> 33.9 | Helped |
| Use model-identified feasible answers instead of human-identified answers for SFT (ablation) | Llama2-7B Clarify DPO w/ Match · AmbigQA Full | Answer F1: 25.9 -> 28.3 | Helped |
| Combine human- and model-identified feasible answers for SFT instead of human data alone (ablation) | Clarify DPO w/ Match · AmbigQA Full | Answer F1: 25.9 -> 32.8; 34.9 -> 36.5; 32.3 -> 33.9 | Helped |
| Merge clarification and answer prediction into one model instead of using separate models (ablation) | Clarify DPO w/ Match · AmbigQA Full | Answer F1: 32.8 -> 28.7; 36.5 -> 34.6; 33.9 -> 31.4 | Hurt |
| Train the selective clarify-or-answer system with double-turn preferences rather than SFT alone | Clarify-or-Direct-Ans · AmbigQA Full | Answer F1: 21.3 -> 24.3; 28.2 -> 29.1; 20.3 -> 24.6 | Helped |
| Use the proposed clarify-or-answer decisions instead of random decisions at the same direct-answer rate | Llama2 Clarify-or-Answer · AmbigQA Full | Direct-Answer Accuracy: 55.4 -> 61.9 | Helped |
| Use the proposed clarify-or-answer decisions instead of random decisions at the same direct-answer rate | Llama3 Clarify-or-Answer · AmbigQA Full | Direct-Answer Accuracy: 55.6 -> 59.2 | Helped |
Surprising findings
- Single-turn RLHF preferences from Starling RM give mixed results, with minor gains or losses on Gemma and Llama3.
- Prompted ProCoT yields degenerate behavior on instruction-tuned Gemma and Llama3, never directly answering.
Limitations
- Only one or two turn interactions are considered; further clarification after the user's answer is not modeled.
- Uses GPT-4 as user simulator for evaluation and only open-domain short factoid QA is tested.
Experimental setup
Datasets: NQ-Open AmbigQA
Benchmarks: AmbigQA
Metrics: Answer F1 number of turns direct-answer accuracy ambiguity accuracy
Baselines: Direct-Ans SFT Clarify SFT Clarify DPO w/ RM (Starling-RM-7B-alpha) Clarify DPO w/ Likelihood ProCoT PPDPP Random
Builds on: DPO ProCoT PPDPP AmbigQA NQ-Open
Closest papers by meaning
- Do LLMs Recognize Your Preferences? Evaluating Personalized Preference Following in LLMs (ICLR 2025 · similarity 0.89)
- Learning to Clarify: Multi-turn Conversations with Action-Based Contrastive Self-Training (ICLR 2025 · similarity 0.88)
- RepLiQA: A Question-Answering Dataset for Benchmarking LLMs on Unseen Reference Content (NeurIPS 2024 · similarity 0.88)
- Putting People in LLMs’ Shoes: Generating Better Answers via Question Rewriter (AAAI 2025 · similarity 0.88)
- AbstentionBench: Reasoning LLMs Fail on Unanswerable Questions (NeurIPS 2025 · similarity 0.88)
- SuRe: Summarizing Retrievals using Answer Candidates for Open-domain QA of LLMs (ICLR 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.