Research map / Large language models / Retrieval and knowledge
DSAS: A Universal Plug-and-Play Framework for Attention Optimization in Multi-Document Question Answering
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes DSAS, a training-free, plug-and-play attention modification for multi-document QA that reweights paragraphs by attention-based relevance and position (CGW) and suppresses attention between key and irrelevant paragraphs (RAS).
Problem
LLMs struggle with multi-document QA due to weak long-range dependency modeling and the lost-in-the-middle problem, and existing fixes either truncate global attention or require costly finetuning.
Approach
Layer-wise attention-based information-flow scores from question and answer-target positions to each paragraph define contextual gate weights, combined with a Gaussian-CDF position-aware weight that boosts central paragraphs; then attention between high-weight and low-weight paragraphs is scaled down, all without training.
Key results
- Average F1 gains of 4.2% on Llama-3.1-8B-Instruct (41.1 to 45.5) and Qwen2.5-14B-Instruct (47.5 to 51.7) across benchmarks.
- Across six models, DSAS improves average F1 by 0.9% to 4.2% over vanilla, and generally outperforms PINE.
- On summarization and code completion LongBench tasks, DSAS improves Qwen2.5-7B average from 36.3 to 39.2 (Rouge-L/Edit Sim).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add DSAS attention optimization | Llama-3.1-8B · HotpotQA, 2WikiMultiHopQA, MuSiQue, and LongBench QA subsets | F1-score (%): +4.2% average F1-score improvement | Helped |
| Add DSAS attention optimization | Qwen2.5-14B · HotpotQA, 2WikiMultiHopQA, MuSiQue, and LongBench QA subsets | F1-score (%): +4.2% average F1-score improvement | Helped |
| Add Contextual Gate Weighting (CGW) (ablation) | DSAS on Multi-doc QA · Multi-doc QA benchmarks | F1-score (%) | Helped |
| Add Reciprocal Attention Suppression (RAS) (ablation) | DSAS on Multi-doc QA · Multi-doc QA benchmarks | F1-score (%) | Helped |
| Add position-aware weighting (ablation) | DSAS on Multi-doc QA · Multi-doc QA benchmarks | F1-score (%) | Helped |
| Set insertion depth to the final 50% of layers rather than the final 25% or 75% (ablation) | DSAS on Llama-3.1-8B · HotpotQA, 2WikiMultiHopQA, MuSiQue, and LongBench | F1-score | Helped |
| Shuffle the paragraph input order (ablation) | DSAS on Qwen2.5-7B, Llama-3.1-8B, and Qwen2.5-14B · HotpotQA, 2WikiMultiHopQA, MuSiQue, and LongBench QA subsets | F1-score (%) | No clear effect |
| Add DSAS | Qwen2.5-7B, Llama-3.1-8B, and Qwen2.5-14B · MuSiQue subsets with 2–4 supporting paragraphs and 10–40 total paragraphs | F1-score | Helped |
Surprising findings
- PINE does not consistently improve over vanilla; it underperforms vanilla on some configurations (e.g., DeepSeek-R1-8B, Qwen2.5-32B LongBench).
- Larger models (Qwen2.5-14B with position-aware weight disabled) achieve competitive results, suggesting larger LLMs mitigate lost-in-the-middle more inherently.
Limitations
- Scalability to extremely long inputs (over 100K tokens) is limited by inherited quadratic attention cost.
- Summarization and code completion evaluation uses simple fixed-length chunking and hand-picked anchors.
- Gains are inconsistent for the smallest model and some benchmarks (e.g., zero gain on 2WikiMQA for Mistral LongBench).
Experimental setup
Datasets: HotpotQA 2WikiMultiHopQA MuSiQue LongBench
Benchmarks: HotpotQA 2WikiMultiHopQA MuSiQue LongBench
Metrics: F1-score
Baselines: Vanilla LLM PINE
Builds on: StreamingLLM PINE
Closest papers by meaning
- Rethinking Generative Large Language Model Evaluation for Semantic Comprehension (ICML 2024 · similarity 0.87)
- Harnessing Large Language Models for Knowledge Graph Question Answering via Adaptive Multi-Aspect Retrieval-Augmentation (AAAI 2025 · similarity 0.87)
- Autofocus Retrieval: An Effective Pipeline for Multi-Hop Question Answering With Semi-Structured Knowledge (NeurIPS 2026 · similarity 0.87)
- Bridging Context Gaps: Leveraging Coreference Resolution for Long Contextual Understanding (ICLR 2025 · similarity 0.86)
- Listening to the Wise Few: Query–Key Alignment Unlocks Latent Correct Answers in Large Language Models (NeurIPS 2026 · similarity 0.86)
- Knowledge Graph Prompting for Multi-Document Question Answering (AAAI 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.