Research map / Vision-language and multimodal / Multimodal LLMs

Nüwa: Mending the Spatial Integrity Torn by VLM Token Pruning

ICLR 2026 · Vision-language and multimodal › Multimodal LLMs · approach: Attention-based visual token pruning · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes Nüwa, a two-stage visual token pruning framework (Boids-inspired separation/alignment/aggregation in the vision encoder, then text-guided pruning inside the LLM) that preserves spatial integrity so VLMs keep grounding performance under heavy token reduction.

Problem

Existing vision token pruning methods preserve VQA performance but degrade severely on visual grounding tasks, because pruning disrupts the global spatial reference frame built from token position embeddings.

Approach

Partition vision tokens into regions, select salient benchmark tokens (CLS attention times key L2-norm), and aggregate neighboring tokens into them with a semantic-similarity times spatial-proximity weight while keeping pillar tokens unchanged and remapping position embeddings to preserve spatial coverage; then prune again inside the LLM using similarity to the averaged text query.

Key results

• On LLaVA-1.5-7B at 64 average tokens (88.9% reduction), Nüwa retains 94.91% of VQA performance versus 93.99% for VisionZip.
• On RefCOCO-series grounding at 64 tokens, Nüwa reaches 47.19% average retention versus 7.28% for VisionZip (RefCOCO-test 29.43 vs 4.04).
• Nüwa adds about 0.01 TFLOPs and 1 ms prefill over VisionZip at 64 tokens, with prefill time of 46 ms vs 124 ms vanilla.

Surprising findings

• Average pooling, a simple baseline, outperforms advanced pruning methods on grounding tasks, and random pruning matches or beats several methods on VQA.
• Blocking last-to-vision attention in layers 8-15 drops RefCOCO accuracy to 2.01 while leaving VQA nearly unchanged.

Limitations

• Grounding gains are measured mainly on LLaVA-1.5-7B; LLaVA-NeXT and Qwen2.5-VL results are reported in the appendix with fewer benchmarks.
• Stage-1 pruning metric adds considerable overhead (about 17.6 MFLOPs vs 4.7 to 9 MFLOPs for baselines), and the method is sensitive to the spatial proximity threshold.
• Numerous inconsistencies across text and tables (e.g., 12 vs 13 datasets, VQA average values differing between tables) reduce confidence in the reported numbers.

Experimental setup

Datasets: LLaVA evaluation suites via lmms-eval

Benchmarks: GQA MMBench MMMU MME VQAv2 TextVQA POPE ScienceQA SEED MM-Vet RefCOCO RefCOCO+

Metrics: accuracy average relative performance retention (%) TFLOPs prefill time (ms)

Baselines: FastV SparseVLM PyramidDrop VisionZip PruMerge+ FEATHER Random pruning Average pooling

Builds on: FastV VisionZip SparseVLM PyramidDrop Boids flocking algorithm

Closest papers by meaning

• QuietPrune: Query-Guided Early Token Pruning for Vision-Language Models (CVPR 2026 · similarity 0.92)
• VLM-Pruner: Buffering for Spatial Sparsity in an Efficient VLM Centrifugal Token Pruning Paradigm (CVPR 2026 · similarity 0.92)
• IVC-Prune: Revealing the Implicit Visual Coordinates in LVLMs for Vision Token Pruning (ICLR 2026 · similarity 0.91)
• IF-Prune: Information-Flow Guided Token Pruning for Efficient Vision-Language Models (CVPR 2026 · similarity 0.91)
• When Token Pruning is Worse than Random: Understanding Visual Token Information in VLLMs (CVPR 2026 · similarity 0.91)
• SwiftVLM: Efficient Vision-Language Model Inference via Cross-Layer Token Bypass (NeurIPS 2026 · similarity 0.90)