Research map / Large language models / Language, safety and interpretability

Exploring Conversational Adaptability: Assessing the Proficiency of Large Language Models in Dynamic Alignment with Updated User Intent

AAAI 2025 · Large language models › Language, safety and interpretability · approach: Faithfulness of LLM explanations · 1 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Introduces the DynDST benchmark (8k dialogues built from MultiWOZ 2.2) and a CoT-free multi-step exact-match evaluator to test whether LLMs align with a user's updated intent in multi-turn dialogue.

Problem

LLM chatbots struggle to adopt a user's updated intent mid-conversation when outdated or incorrect context remains in the history, and existing datasets and evaluators are unsuited to measuring this for open-ended LLM outputs.

Approach

Synthetically inject a false slot value and a later correction turn into MultiWOZ 2.2 dialogues, then score LLM replies with a prompt-free, CoT-free multi-step exact-match pipeline (exact match, canonical mapping, stop-word removal, ROUGE, edit distance) that decides whether the reply contains the new or old intent.

Key results

• GPT-3.5 (gpt-3.5-turbo-0125) reaches 73.7% Align in the best single run of Exp. 2 and 80.1% with 5-run majority voting, while GPT-4 reaches 70.3% and 76.0% under the same setting.
• Injecting explicit negation plus a mock bot update (Exp. 2) raises Align over the implicit update baseline (Exp. 1) for every model, e.g. Vicuna-33B improves by 17.5 points.
• Exp. 3 (swapping false and update turns) yields only 5.4-7.2% Align for GPT-3.5, showing models mostly follow the most recent turn.

Measured effects reported in the paper

Surprising findings

• GPT-3.5 outperforms GPT-4 on the dynamic alignment task, which the authors attribute to possible safety training in GPT-4.
• Gemini's single-turn input format performs on par with or slightly better than GPT-3.5's multi-turn format.

Limitations

• Evaluation relies on string-based exact match plus ROUGE heuristics, so it may miss semantically correct answers phrased differently.
• Dataset is synthetic, built by editing MultiWOZ 2.2 slot values in a restricted set of travel-booking domains.
• Results are at temperature 0 and only a small set of models was tested; GPT-4 ablations were omitted due to cost.

Experimental setup

Datasets: MultiWOZ 2.2

Benchmarks: DynDST

Metrics: Align accuracy No Align rate N.A. rate upper bound with multiple runs

Baselines: GPT-4 Gemini 1.0 Pro Vicuna-33B Vicuna-13B Vicuna-7B Llama-2-13B-chat Llama-2-7B-chat

Builds on: MultiWOZ 2.2 knowledge editing (MEMIT) dialogue contradiction detection (DECODE, CDConv)

Closest papers by meaning

• Beyond Prompts: Dynamic Conversational Benchmarking of Large Language Models (NeurIPS 2024 · similarity 0.89)
• Communication Accommodation Between Large Language Models and Users Across Cultures (Student Abstract) (AAAI 2025 · similarity 0.87)
• LongMemEval: Benchmarking Chat Assistants on Long-Term Interactive Memory (ICLR 2025 · similarity 0.87)
• Mutable Transcripts: Mitigating Context Pollution through Editable Conversation State (NeurIPS 2026 · similarity 0.86)
• A Statistical Framework for Ranking LLM-based Chatbots (ICLR 2025 · similarity 0.86)
• Learning to Clarify: Multi-turn Conversations with Action-Based Contrastive Self-Training (ICLR 2025 · similarity 0.86)