Research map / Architectures and efficiency / Attention and transformers
StreamingDialogue: Prolonged Dialogue Learning via Long Context Compression with Minimal Losses
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes StreamingDialogue, which compresses dialogue history into End-of-Utterance "conversational attention sinks" with attention masking and two self-learning objectives (SMR and LMR) to enable prolonged dialogue with lower cost than dense attention.
Problem
Standard LLMs struggle with long multi-turn dialogues due to quadratic attention cost and loss of historical information (consistency) when using sparse or sliding-window attention.
Approach
Observes that End-of-Utterance tokens aggregate attention like StreamingLLM's initial-token sinks; caches only the first token, EoU sink tokens, and the most recent two utterances, and trains with short-memory reconstruction (SMR) and long-memory reactivation (LMR) objectives under a custom attention mask to make EoU tokens compress utterance content.
Key results
- On MSC, StreamingDialogue raises B-avg from 16.76% (StreamingLLM) to 19.33% and R-L from 14.21% to 15.86%.
- Reconstructing dialogue from only the conv-attn sink of each utterance achieves BLEU-1 of 89.19%.
- Relative to dense attention with recomputation, the method reports a 4x per-token latency speedup and 18x memory reduction at input length 2,048.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use conv-attn sink attention instead of StreamingLLM attention | Llama-2-7B StreamingDialogue · MSC test | BLEU-avg: 16.76 -> 19.33 | Helped |
| Use conv-attn sink attention instead of StreamingLLM attention | Llama-2-7B StreamingDialogue · MSC test | ROUGE-L: 14.21 -> 15.86 | Helped |
| Use conv-attn sink attention instead of StreamingLLM attention | Llama-2-7B StreamingDialogue · PersonaChat test | Distinct-2: 32.64% to 37.23% | Helped |
| Use conv-attn sink attention instead of StreamingLLM attention, without training | Llama-2-7B-Chat · MSC test | BLEU-avg: 20.16 -> 20.19 | Helped |
| Use conv-attn sink attention instead of StreamingLLM attention, without training | Llama-3-8B-Instruct · MSC test | BLEU-avg: 16.48 -> 16.77 | Helped |
| Use conv-attn sink attention instead of StreamingLLM attention, without training | Mistral-7B · MSC test | BLEU-avg: 12.75 -> 13.33 | Helped |
| Add SMR and LMR (ablation) | StreamingDialogue · MSC test | BLEU-avg: 17.32 -> 19.33 | Helped |
| Add SMR and LMR (ablation) | StreamingDialogue · MSC test | ROUGE-L: 10.25 -> 15.86 | Helped |
| Add SMR and LMR (ablation) | StreamingDialogue · MSC test | Distinct-3: 46.15 -> 50.27 | Helped |
| Add SMR (ablation) | StreamingDialogue · MSC test | BLEU-avg: 18.87 -> 19.33 | Helped |
| Add SMR (ablation) | StreamingDialogue · MSC test | ROUGE-L: 15.66 -> 15.86 | Helped |
| Add LMR (ablation) | StreamingDialogue · MSC test | BLEU-avg: 18.25 -> 19.33 | Helped |
Surprising findings
- The conv-attn sink phenomenon appears in BERT with 79.86% of heads, versus 14.36% for Llama2 and 16.32% for GLM.
- Without any training, the method still outperforms StreamingLLM on MSC for Llama-2-7B-Chat, Llama-3-8B-Instruct, and Mistral-7B.
Limitations
- SMR and LMR training adds computational cost (about two hours on two A100-40G GPUs).
- Selective caching of only key conv-attn sinks is left for future work; dialogue structure use is limited.
- Evaluated only on dialogue datasets and a limited set of structured texts.
Experimental setup
Datasets: PersonaChat MSC (Multi-Session Chat) Topical-Chat MultiWOZ
Benchmarks: PersonaChat MSC Topical-Chat MultiWOZ
Metrics: perplexity BLEU ROUGE Distinct USL-H Dial-M C score
Baselines: Dense attention Local attention Big Bird StreamingLLM MemBART HRED VHRED
Builds on: StreamingLLM Longformer Big Bird MemBART
Closest papers by meaning
- Efficient Streaming Language Models with Attention Sinks (ICLR 2024 · similarity 0.90)
- Compressed Context Memory for Online Language Model Interaction (ICLR 2024 · similarity 0.88)
- Efficient Prompt Compression with Evaluator Heads for Long-Context Transformer Inference (NeurIPS 2025 · similarity 0.86)
- FastLongSpeech: Enhancing Large Speech-Language Models for Efficient Long-Speech Processing (NeurIPS 2025 · similarity 0.86)
- Prompt Compression with Context-Aware Sentence Encoding for Fast and Improved LLM Inference (AAAI 2025 · similarity 0.86)
- SeCom: On Memory Construction and Retrieval for Personalized Conversational Agents (ICLR 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.