Research map / Large language models / Language, safety and interpretability

Found in Conversation: LLMs Teach Themselves to Close the Multi-Turn Gap

NeurIPS 2026 · Large language models › Language, safety and interpretability · approach: LLM unlearning and benchmarks · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes Found in Conversation (FiC): SFT warm-start on sharded multi-turn GSM8K with deferral responses, then View-Asymmetric Self-Distillation that distills a single-turn concatenated-view teacher into the same model's multi-turn student view, closing much of the Lost-in-Conversation gap.

Problem

LLMs perform far worse when task information is revealed gradually across underspecified multi-turn turns than in a single fully specified prompt (Lost-in-Conversation), and existing fixes (prompting, SFT, RL, external-teacher distillation) have difficulties.

Approach

Self-distillation with input-view asymmetry: the same frozen backbone acts as teacher on an information-equivalent single-turn concatenated view and as student on the multi-turn sharded view; a token-level JS divergence on the final answer aligns them, with on-policy intermediate turns, and no external teacher or privileged information.

Key results

• On the GSM8K sharded setting, FiC reaches 100% recovery rate on Llama-3.2-3B and Llama-3.1-8B and 92-94% on Qwen2.5-7B, OLMo-2-13B, and Phi-4-14B (e.g., Llama-3.1-8B sharded 44.7 -> 71.7).
• FiC reduces mean tokens per sharded conversation by up to 32.5% (Qwen2.5-7B) while keeping turn counts unchanged.
• Vanilla distillation from Llama-3.1-8B to Llama-3.2-3B drops accuracy to 30.7 from a 39.2 baseline, while FiC reaches 60.4.

Surprising findings

• Vanilla distillation from a stronger teacher (Llama-3.1-8B to Llama-3.2-3B) hurts performance rather than helping.
• Inference-time interventions (prompting, judge gating) often fail to improve or degrade sharded accuracy.

Limitations

• User simulator is cooperative and non-pivoting; does not model corrections or retractions.
• Evaluation test splits are small (~100 instances per task) limiting resolution.
• Single-turn accuracy regresses notably for Llama-3.2-3B (60.2 to 52.6 on concat).

Experimental setup

Datasets: GSM8K Spider Berkeley Function Calling Leaderboard Lost-in-Conversation sharded benchmark

Benchmarks: GSM8K sharded (Lost-in-Conversation) Spider sharded (Database) BFCL sharded (Actions)

Metrics: accuracy recovery rate mean tokens per conversation

Baselines: Lost-in-Conversation sharded baseline prompt-rule prompt-selfcheck gated-external gated-self vanilla distillation SFT only off-policy VASD

Builds on: Lost-in-Conversation on-policy distillation self-distillation with privileged information LoRA

Closest papers by meaning

• MINT: Evaluating LLMs in Multi-turn Interaction with Tools and Language Feedback (ICLR 2024 · similarity 0.90)
• Learning to Clarify: Multi-turn Conversations with Action-Based Contrastive Self-Training (ICLR 2025 · similarity 0.89)
• Toward Self-Improvement of LLMs via Imagination, Searching, and Criticizing (NeurIPS 2024 · similarity 0.88)
• Regressing the Relative Future: Efficient Policy Optimization for Multi-turn RLHF (ICLR 2025 · similarity 0.88)
• PRISM: Demystifying Retention and Interaction in Mid-Training (ICML 2026 · similarity 0.88)
• A Closer Look at the Limitations of Instruction Tuning (ICML 2024 · similarity 0.88)