Research map / Vision-language and multimodal / Multimodal LLMs

DUET-VLM: Dual stage Unified Efficient Token reduction for VLM Training and Inference

CVPR 2026 · Vision-language and multimodal › Multimodal LLMs · approach: Attention-based visual token pruning · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes DUET-VLM, a dual-stage visual token reduction framework that merges vision-encoder tokens via attention-guided local clustering and then drops visual tokens layer-wise in the LLM using salient text-to-vision attention, with training-aware variants for LLaVA and Video-LLaVA.

Problem

VLMs are expensive because dense visual tokenization produces hundreds to thousands of visual tokens; existing methods either merge too early or prune too uniformly, losing accuracy under aggressive compression.

Approach

Combine vision-only redundancy-aware merging (dominant tokens plus local-cluster contextual tokens selected by CLIP V2V attention) with layer-wise text-guided pruning inside the language backbone, using salient text tokens' cross-attention to rank visual tokens, and train end-to-end under the same compression.

Key results

• On LLaVA-1.5-7B inference-only, DUET-VLM (C) retains 99.0% of baseline average accuracy with 192 tokens (67% reduction) and 95.4% at 64 tokens (89% reduction), above VisionZip's 97.7% and 92.8%.
• With training, DUET-VLM (C+S) on LLaVA-1.5-7B reaches 99.7% at 192 tokens and 97.6% at 64 tokens relative to the baseline.
• On Video-LLaVA-7B, DUET-VLM (C) reaches 100.8% of baseline at 960 tokens (53.1% reduction) and 97.6% at 136 tokens (93.4% reduction), versus 91.0% for VisionZip at 136 tokens.

Surprising findings

• Pruming all visual tokens after the 24th layer of LLaVA-1.5 has little effect on accuracy, suggesting late-layer visual tokens are redundant.
• Video-LLaVA with DUET-VLM slightly exceeds the full-token baseline at 960 tokens.

Limitations

• Speed analysis is limited; the paper says exact inference and training times are not reported and kernels are not optimized.
• Video training and longer-horizon video evaluation are left for future work.
• Vision-side reduction relies on CLIP V2V self-attention, which may be imprecise for region selection.

Experimental setup

Datasets: POPE GQA TextVQA MME ScienceQA-Image SeedBench-Image TGIF-QA MSVD-QA MSRVTT-QA

Benchmarks: POPE SQA-Image TextVQA MME GQA SeedBench-Image TGIF-QA MSVD-QA MSRVTT-QA

Metrics: accuracy relative average accuracy (% of baseline) speedup

Baselines: FastV HiRED FitPrune PruMerge SparseVLM PyramidDrop VisionZip

Builds on: VisionZip PyramidDrop

Closest papers by meaning

• ApET: Approximation-Error Guided Token Compression for Efficient VLMs (CVPR 2026 · similarity 0.93)
• Variation-aware Vision Token Dropping for Faster Large Vision-Language Models (CVPR 2026 · similarity 0.93)
• Inference Optimal VLMs Need Fewer Visual Tokens and More Parameters (ICLR 2025 · similarity 0.93)
• Hybrid Token Compression for Vision-Language Models (CVPR 2026 · similarity 0.93)
• SparseVLM: Visual Token Sparsification for Efficient Vision-Language Model Inference (ICML 2025 · similarity 0.92)
• VisionTrim: Unified Vision Token Compression for Training-Free MLLM Acceleration (ICLR 2026 · similarity 0.92)