Research map / Large language models / Language, safety and interpretability

SayNext-Bench: Why Do LLMs Struggle with Next-Utterance Anticipation?

NeurIPS 2026 · Large language models › Language, safety and interpretability · approach: Statistical frameworks for LLM evaluation · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Introduces SayNext-Bench, SayNext-PC dataset, and SayNext-Chat, a dual-route MLLM with learnable priming tokens, to test and improve next-utterance anticipation from multimodal dialogue cues.

Problem

Current LLMs/MLLMs struggle to anticipate what a human speaker will say next from multimodal cues (gesture, gaze, emotion) in dialogue, and existing benchmarks and metrics do not capture this one-to-many task.

Approach

A cognitively inspired dual-route MLLM (SayNext-Chat) with a fast route (visual+text to response) and a deep route that predicts priming factors via learnable priming tokens injected into the LLM, trained with an adaptive-weighted priming loss; evaluated with a three-level metric framework.

Key results

• SayNext-Chat achieves BLEU-4 lexical overlap of 2.307% and ROUGE-L 17.957 on SayNext-PC2K subject-dependent, versus 1.081 and 14.623 for the best baseline GPT-4o.
• On LLM-as-judge Level 3, SayNext-Chat is top-1 at 29.93% (GPT-4.1) and 32.00% (Gemini-2.5-Pro), versus 21.78% and 25.19% for InternVL2.
• In the user study, SayNext-Chat is selected as most similar in 34.25% of cases, above GPT-4o (26.92%) and InternVL2 (24.58%).

Surprising findings

• Prompt-based injection of priming factors or vectors degrades performance relative to the fine-tuned baseline, biasing models toward numeric cues.
• The no-vision baseline still outperforms several vision-equipped baselines on discrete emotion and intention consistency.

Limitations

• Lexical overlap remains very low (below 20% for most models), so the task remains hard even with the proposed method.
• Data is restricted to post-match tennis press conferences, a constrained social setting; generalization to open-ended dialogue is untested.
• Noisy samples with modality inconsistencies or off-topic content can hinder learning.

Experimental setup

Datasets: SayNext-PC2K SayNext-PC19K IEMOCAP

Benchmarks: SayNext-PC2K SayNext-PC19K IEMOCAP (cross-scenario)

Metrics: BLEU-4 ROUGE-L BERTScore-F1 Sentence-BERT similarity NRC-VAD valence/arousal consistency discrete emotion consistency intention consistency LLM-as-judge preference user study selection rate

Baselines: GPT-4o Gemini-2.5 InternVL2-8B VideoLLaMA3-7B LLaVA-NeXT-Video-7B InstructBLIP-7B Emotion-LLaMA-7B

Builds on: LoRA semantic priming iMiGUE

Closest papers by meaning

• SD-Eval: A Benchmark Dataset for Spoken Dialogue Understanding Beyond Words (NeurIPS 2024 · similarity 0.89)
• Talking Turns: Benchmarking Audio Foundation Models on Turn-Taking Dynamics (ICLR 2025 · similarity 0.87)
• Paralinguistics-Aware Speech-Empowered Large Language Models for Natural Conversation (NeurIPS 2024 · similarity 0.87)
• FutureOmni: Evaluating Future Forecasting from Omni-Modal Context for Multimodal LLMs (ICML 2026 · similarity 0.86)
• A Comprehensive Analysis of the Effectiveness of Large Language Models as Automatic Dialogue Evaluators (AAAI 2024 · similarity 0.86)
• HumanSense: From Multimodal Perception to Empathetic Context-Aware Responses Through Reasoning MLLMs (AAAI 2026 · similarity 0.86)