Research map / Vision-language and multimodal / Multimodal LLMs

URaG: Unified Retrieval and Generation in Multimodal LLMs for Efficient Long Document Understanding

AAAI 2026 · Vision-language and multimodal › Multimodal LLMs · approach: Unified any-modality language models · 1 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes URaG, which turns early Transformer layers of a Qwen2.5-VL MLLM into a lightweight cross-modal evidence selector that keeps top-k pages and discards the rest, unifying retrieval and generation in one model for efficient long document QA.

Problem

Multimodal LLMs struggle with long multi-page document understanding due to information interference from irrelevant pages and quadratic Transformer cost; existing token compression loses detail and external retrievers add complexity and prevent end-to-end optimization.

Approach

Observes that early MLLM layers attend broadly while deeper layers focus on evidence pages, then uses early-layer hidden states with a two-linear-layer retrieval head and contextualized late interaction to select top-k pages mid-forward-pass, dropping other pages' visual tokens for deeper layers.

Key results

• URaG-7B reaches 88.2 on MPDocVQA, 57.6 on DUDE, 72.1 on SlideVQA and 52.2 on LongDocURL, outperforming Qwen2.5-VL-7B on those benchmarks.
• Reduces FLOPs by 44.0% to 55.8% versus the baseline when input grows from 20 to 100 pages (URaG-7B, SlideVQA setting).
• On MMLongBench-Doc, URaG-7B achieves 33.8 generalized accuracy vs 26.2 for Qwen2.5-VL-7B; retrieval module adds only 0.05-0.07% of parameters.

Surprising findings

• Inserting the pretrained retrieval module without any MLLM fine-tuning (URaG-3B w/o finetune) already surpasses the fully fine-tuned baseline on SlideVQA and MMLongBench-Doc.
• Fine-tuning the baseline degrades performance on LongDocURL (40.0 to 37.3), which the authors attribute to overfitting or domain mismatch.

Limitations

• Uses a fixed top-k (k=5) retrieval strategy that may omit dispersed evidence or include redundant pages.
• Analysis and training are limited to a small set of benchmarks and two MLLM backbones (Qwen2.5-VL, InternVL2.5).

Experimental setup

Datasets: MPDocVQA DUDE SlideVQA

Benchmarks: SlideVQA MMLongBench-Doc DUDE MPDocVQA LongDocURL

Metrics: Top1 accuracy Top5 accuracy ANLS exact match generalized accuracy F1 FLOPs

Baselines: BM25 SBERT BGE-large NV-Embed-v2 ColPali MM-Embed SV-RAG M3DocRAG mPLUG-DocOwl2 CREAM GRAM Qwen2.5-VL

Builds on: ColBERT-style contextualized late interaction Qwen2.5-VL SV-RAG LoRA

Closest papers by meaning

• DocSeeker: Structured Visual Reasoning with Evidence Grounding for Long Document Understanding (CVPR 2026 · similarity 0.90)
• Resolving Evidence Sparsity: Agentic Context Engineering for Long-Document Understanding (CVPR 2026 · similarity 0.89)
• DocLayLLM: An Efficient Multi-modal Extension of Large Language Models for Text-rich Document Understanding (CVPR 2025 · similarity 0.88)
• SV-RAG: LoRA-Contextualizing Adaptation of MLLMs for Long Document Understanding (ICLR 2025 · similarity 0.88)
• Needle In A Multimodal Haystack (NeurIPS 2024 · similarity 0.88)
• SLQ: Bridging Modalities via Shared Latent Queries for Retrieval with Frozen MLLMs (ICML 2026 · similarity 0.87)