Research map / Vision-language and multimodal / Multimodal LLMs

Enhancing Multi-Image Understanding through Delimiter Token Scaling

ICLR 2026 · Vision-language and multimodal › Multimodal LLMs · approach: VLM capability analysis · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Scales hidden states of image delimiter tokens in LVLMs (training-free) to reduce cross-image information leakage and improve multi-image and text-only multi-document/multi-table performance.

Problem

LVLMs degrade on multi-image inputs due to cross-image information leakage, and existing image delimiter tokens do not fully block cross-image attention.

Approach

Analyze image delimiter tokens as attention sinks that act as image-specific tags; multiply their hidden states by a factor lambda>1 at early layers, which strengthens intra-image attention and suppresses cross-image attention without changing the attention kernel.

Key results

• On Mantis, Qwen2.5-VL-3B improves from 59.91 to 63.13 accuracy with delimiter token scaling, with no extra memory or inference time.
• On MuirBench, Qwen2.5-VL-3B improves from 37.31 to 42.42.
• Qwen2.5-3B with the method (TQABench 37.84) outperforms the Qwen2.5-7B baseline (37.50) on TQABench.

Surprising findings

• Replacing delimiter tokens with other special tokens such as <|im_start|> causes a drop in performance, and scaling <|im_start|> alone drops accuracy below baseline (53.00 vs 59.91).
• Scaling factors below 1 reduce performance below baseline and leave cross-image leakage present.

Limitations

• Not applicable to videos or inputs lacking explicit frame-separating tokens.
• Requires access to hidden states, so only applicable to open-weight models.
• Hyperparameters (scaling layer and factor) tuned per model; text-only tasks tuned directly on the test set.

Experimental setup

Datasets: Mantis-Eval MuirBench MIRB Q-Bench2 MultiNews WCEP-10 TQABench OKVQA TextVQA VizWiz

Benchmarks: Mantis-Eval MuirBench MIRB Q-Bench2 MultiNews WCEP-10 TQABench OKVQA VizWiz

Metrics: accuracy ROUGE-1 ROUGE-2 ROUGE-L peak VRAM inference time

Baselines: FOCUS M-RoPE temporal embedding First token scaling Q scaling K scaling V scaling

Builds on: FlashAttention M-RoPE attention sink

Closest papers by meaning

• Enhancing Multimodal Large Language Models Complex Reason via Similarity Computation (AAAI 2025 · similarity 0.89)
• Rethinking Visual Token Reduction in LVLMs Under Cross-Modal Misalignment (AAAI 2026 · similarity 0.89)
• Variation-aware Vision Token Dropping for Faster Large Vision-Language Models (CVPR 2026 · similarity 0.89)
• The Narrow Gate: Localized Image-Text Communication in Native Multimodal Models (NeurIPS 2025 · similarity 0.89)
• Learning Interleaved Image-Text Comprehension in Vision-Language Large Models (ICLR 2025 · similarity 0.88)
• Benchmarking Large Vision-Language Models on Fine-Grained Image Tasks: A Comprehensive Evaluation (ICLR 2026 · similarity 0.88)