Research map / Large language models / Alignment and preferences
Learning to Clarify: Multi-turn Conversations with Action-Based Contrastive Self-Training
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes ACT, a quasi-online DPO variant that builds action-contrastive preference pairs and uses on-policy sampling with user-simulated trajectories to tune LLMs for clarification-asking in multi-turn conversations, evaluated on PACIFIC, Abg-CoQA, and the new AmbigSQL.
Problem
LLM-based conversational agents overhedge or guess user intent instead of asking clarifying questions, and tuning for multi-turn disambiguation is hard when task-specific conversation data is scarce.
Approach
Contrastive action-based preference pairs (clarify vs. answer) where the losing response is on-policy sampled from the current policy and, if it chooses the right action, its simulated multi-turn trajectory is scored to decide winner/loser; the DPO objective is then applied to these quasi-online pairs.
Key results
- On PACIFIC with 50 conversations, ACT reaches 82.2 Macro F1 vs 69.0 for SFT (up to 19.1% relative improvement).
- On AmbigSQL with 50 conversations, ACT achieves 43.6 Execution Match vs 21.9 for SFT.
- Zephyr tuned with ACT on PACIFIC with zero-shot Gemini 1.5 Pro action labels (98.5% agreement) performs nearly identically to crowdsourced labels.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| tune with ACT instead of SFT | Zephyr 7B-β · PACIFIC public evaluation, 50 conversations | Macro F1: 69.0 -> 82.2 | Helped |
| tune with ACT instead of SFT | Zephyr 7B-β · PACIFIC public evaluation, 50 conversations | Post-Clarify F1: 43.5 -> 57.2 | Helped |
| tune with ACT instead of SFT | Zephyr 7B-β · Abg-CoQA test, 50 conversations | Traj. Similarity: 64.2 -> 68.8 | Helped |
| tune with ACT instead of SFT | Zephyr 7B-β · AmbigSQL test, 50 conversations | Execution Match: 21.9 -> 43.6 | Helped |
| tune with ACT instead of IRPO | Zephyr 7B-β · AmbigSQL test, 50 conversations | Accuracy: 91.0 -> 80.8 | Hurt |
| use crowdsourced action labels instead of pseudo-labels | ACT on PACIFIC · PACIFIC, 50 conversations | Macro F1: 80.1 -> 82.2 | Helped |
| randomly sample both winning and losing actions instead of action-based preferences (ablation) | ACT on PACIFIC · PACIFIC, 50 conversations | Macro F1: 82.2 -> 63.2 | Hurt |
| add on-policy sampling (ablation) | ACT on PACIFIC · PACIFIC, 50 conversations | Macro F1: 74.8 -> 82.2 | Helped |
| add on-policy sampling (ablation) | ACT on PACIFIC · PACIFIC, 50 conversations | Traj. F1: 59.1 -> 61.9 | Helped |
| add trajectory simulation (ablation) | ACT on PACIFIC · PACIFIC, 50 conversations | Post-Clarify F1: 50.1 -> 57.2 | Helped |
| replace SFT with ACT (ablation) | Gemma 2B · PACIFIC, 50 conversations | Macro F1: 57.7 -> 62.7 | Helped |
| replace SFT with ACT (ablation) | Mistral 7B · PACIFIC, 50 conversations | Traj. F1: 51.4 -> 57.6 | Helped |
Surprising findings
- Off-policy DPO-Dist and IRPO do not consistently beat SFT; e.g., DPO-Dist (Pro v. Flash) lowers Macro F1 on PACIFIC at 100 and 250 conversations.
- Many-shot ICL with Gemini Pro (50 to 250 examples) does not improve implicit action recognition uniformly on Abg-CoQA.
Limitations
- Relies on task-specific heuristics and commercial LLMs (Gemini) for action classification, user simulation, and preference data generation.
- Limited online exploration because tasks have objective right/wrong answers, so sampled trajectories may match offline data.
- Action-level performance on Abg-CoQA is inconsistent, with SFT sometimes outperforming ACT, linked to low inter-annotator agreement.
Experimental setup
Datasets: PACIFIC Abg-CoQA AmbigSQL Spider CoSQL UltraChat UltraFeedback
Benchmarks: PACIFIC Abg-CoQA AmbigSQL
Metrics: Macro F1 Turn F1 Trajectory F1 Post-Clarify F1 Turn Similarity Trajectory Similarity SentenceBERT semantic similarity Execution Match PC Execution Match Action Accuracy
Baselines: Standard ICL Chain-of-thought prompting Proactive MIPrompt SFT IRPO DPO-Dist ACT w/o on-policy sampling
Builds on: Direct Preference Optimization Iterative Reasoning Preference Optimization Spider Zephyr
Closest papers by meaning
- Found in Conversation: LLMs Teach Themselves to Close the Multi-Turn Gap (NeurIPS 2026 · similarity 0.89)
- Enhancing Personalized Multi-Turn Dialogue with Curiosity Reward (NeurIPS 2025 · similarity 0.89)
- Decoupling Conversational Dynamics in Full-Duplex Spoken Models through Reinforcement Learning (NeurIPS 2026 · similarity 0.88)
- Modeling Future Conversation Turns to Teach LLMs to Ask Clarifying Questions (ICLR 2025 · similarity 0.88)
- Aligning Spoken Dialogue Models from User Interactions (ICML 2025 · similarity 0.88)
- DialogXpert: Driving Intelligent and Emotion-Aware Conversations Through Online Value-Based Reinforcement Learning with LLM Priors (AAAI 2026 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.