Research map / Vision-language and multimodal / Multimodal LLMs
LV-XAttn: Distributed Cross-Attention for Long Visual Inputs in Multimodal Large Language Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes LV-XAttn, a distributed exact cross-attention scheme for MLLMs that keeps large visual key-value blocks local and rotates small query blocks across GPUs, plus activation recomputation of visual features to fit longer visual inputs.
Problem
Cross-attention over very large visual token sets (e.g., long videos) in MLLMs has high memory demand and, when distributed with Ring Attention, becomes communication-bound, consuming up to 88% of iteration time.
Approach
Exploit the asymmetry S_Q << S_KV in MLLM cross-attention: keep K/V shards local on each worker and rotate query, partial output, and softmax statistics blocks instead of K/V, overlapping this communication with computation; additionally recompute K/V from a single saved copy of visual features during backward to save memory.
Key results
- LV-XAttn achieves up to 10.62x end-to-end per-iteration speedup over Ring Attention for Llama 3-V-11b on 16 A100 80GB GPUs, and up to 27.59x cross-attention speedup.
- On 8 A30 GPUs, cross-attention speedups reach 45.85x for mPLUG-Owl3-7b, with end-to-end speedups of 1.04-3.45x across models.
- Activation recomputation enables 1.5x-1.6x more frames with less than 8% runtime overhead; LV-XAttn overhead is under 0.42% versus a no-communication baseline.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use LV-XAttn (keep KV local, rotate queries/outputs) instead of Ring Attention for cross-attention | cross-attention layers in MLLMs · 16 A100 80GB, Llama 3-V-11b and mPLUG-Owl3/OpenFlamingo models | End-to-end iteration speedup: up to 10.62× total (Llama 3-V); 1.92× for OpenFlamingo-3b | Helped |
| Use LV-XAttn instead of Ring Attention for cross-attention | cross-attention layers · 8 A30 24GB, slower interconnect | Cross-attention speedup: 22.43× to 33.77× (Llama 3-V); up to 45.85× (mPLUG-Owl3-7b) | Helped |
| Use LV-XAttn instead of Ring Attention for cross-attention | cross-attention layers · 8 A30 24GB, total iteration time | End-to-end speedup: total speedup 1.04× to 3.45× across models; larger CA speedups do not translate into larger total speedup | Mixed |
| Use LV-XAttn instead of DeepSpeed-Ulysses for cross-attention | mPLUG-Owl3-2b on A100 · Per-iteration time, 12/6/3 GPUs | Speedup vs DeepSpeed-Ulysses: 1.34–1.55× speedup reported in text | Helped |
| Use LV-XAttn instead of DeepSpeed-Ulysses (head parallelism) | OpenFlamingo-3b on A30 · 8 and 4 GPUs, long text and frames | Feasibility (OOM vs run): Ulysses OOM where LV-XAttn runs; LV-XAttn processes >4× longer inputs | Helped |
| Overlap communication with computation in LV-XAttn (ablation) | OpenFlamingo-3b cross-attention · 6 A100 40GB, 2048 frames per worker | Overhead vs no-communication baseline: overhead less than 0.42% with overlap | Helped |
| Overlap communication with computation in Ring Attention (ablation) | Ring Attention cross-attention baseline · 6 A100 40GB, OpenFlamingo-3b | Cross-attention runtime: reduces runtime, but effect limited because KV communication cannot be fully hidden | Helped |
| Activation recomputation of KV from a single shared visual feature copy in cross-attention layers (ablation) | mPLUG-Owl-7b and OpenFlamingo-3b · 3 A100 40GB / A30 24GB single node | Max frames processable: 1.5× more frames (mPLUG-Owl-7b) and 1.6× (OpenFlamingo-3b), with under 8% runtime overhead | Helped |
| Use LV-XAttn instead of Ring Attention for cross-attention on OpenFlamingo-3b with dense cross-attention layers | OpenFlamingo-3b, 16 A100 · 64K text, 64K frames | Total iteration speedup: 1.36× to 1.92× total speedups; OpenFlamingo-9b 1.16× to 1.4× | Mixed |
Surprising findings
- For OpenFlamingo-9b, LV-XAttn's end-to-end speedup over Ring Attention is modest (1.16-1.4x on A100), since cross-attention is a smaller share of total time there.
Limitations
- Speedups are measured on randomly generated inputs of specified sizes rather than end-to-end task accuracy.
- Head-parallel and Ring baselines are compared mainly on runtime; the approach is most beneficial when S_KV >> S_Q and less so for self-attention.
- Gains for OpenFlamingo-9b end-to-end are modest because cross-attention is a smaller share of its runtime.
Experimental setup
Datasets: Video-MME
Benchmarks: Video-MME
Metrics: per-iteration wall-clock time cross-attention time speedup communication volume
Baselines: Ring Attention DeepSpeed-Ulysses
Builds on: Ring Attention FlashAttention DeepSpeed-Ulysses
Closest papers by meaning
- Q Cache: Visual Attention Is Valuable in Less than Half of Decode Layers for Multimodal Large Language Model (AAAI 2026 · similarity 0.89)
- Conical Visual Concentration for Efficient Large Vision-Language Models (CVPR 2025 · similarity 0.88)
- Boosting Multimodal Large Language Models with Visual Tokens Withdrawal for Rapid Inference (AAAI 2025 · similarity 0.88)
- ST3: Accelerating Multimodal Large Language Model by Spatial-Temporal Visual Token Trimming (AAAI 2025 · similarity 0.88)
- VISion On Request: Enhanced VLLM efficiency with sparse, dynamically selected, vision-language interactions (CVPR 2026 · similarity 0.87)
- Where Does Vision Meet Language? Understanding and Refining Visual Fusion in MLLMs via Contrastive Attention (CVPR 2026 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.